A data labeling method, device, equipment and computer-readable storage medium
By automatically determining the labeling area and building the labeling data, the problem of low efficiency of manual labeling data in the target detection task in the prior art is solved, efficient and accurate data labeling is achieved, and labor costs are reduced.
Patent Information
- Application Number
- CN202110835425.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-07-23
AI Technical Summary
In the prior art, the target detection task requires a large amount of manual labeling data, resulting in low data labeling efficiency and high labor cost, and different labeling personnel have different dimensions, making it difficult to ensure the quality of labeling data.
Provide a data labeling method and device, by obtaining the image to be marked and the target labeling category, automatically determine the labeling area, and determine the target labeling area from the image based on template matching, target tracking, motion target detection, etc., to construct the labeling data. Reduce manual intervention.
Improve the efficiency of data annotation, reduce the time and labor cost of labeling, and ensure the quality and consistency of labeling data.
Smart Images

Figure CN113822137B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer vision technology in the field of artificial intelligence, and in particular to a data labeling method, device, equipment and computer-readable storage medium. Background Art
[0002] Object detection is an important application direction in artificial intelligence, which aims to find the location and category of the object of interest from the image. At present, the object detection task is mostly implemented by neural network models, but neural network models need a large amount of labeled data when learning to achieve good results.
[0003] However, in the related art, when generating labeled annotation data, annotators are still required to manually annotate each image, which takes a long time, resulting in low efficiency of data annotation. Summary of the invention
[0004] The embodiments of the present application provide a data labeling method, apparatus, device and computer-readable storage medium, which can improve the efficiency of data labeling.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a data labeling method, including:
[0007] Obtaining an image to be annotated and a target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated;
[0008] For the image to be annotated, determining a corresponding annotation area determination method;
[0009] Based on the annotation area determination method, determining a target annotation area where an object corresponding to the target annotation category is located from the image to be annotated;
[0010] The annotation data is constructed according to the target annotation area of the image to be annotated and the target annotation category.
[0011] The present application provides a data labeling device, including:
[0012] A data acquisition module, used to acquire the image to be annotated and the target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated;
[0013] A marking method determination module, used to determine a corresponding marking area determination method for the image to be marked;
[0014] A data annotation module, configured to determine, based on the annotation area determination method, a target annotation area where an object corresponding to the target annotation category is located from the image to be annotated;
[0015] The data construction module is used to construct the annotation data according to the target annotation area of the image to be annotated and the target annotation category.
[0016] In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on template matching; the data annotation module is also used to match the candidate object from the image to be annotated for the template object corresponding to the target annotation category obtained; and determining the image area where the candidate object is located as the target annotation area to achieve the determination of the annotation area based on template matching.
[0017] In some embodiments of the present application, the image to be annotated is a video frame sequence, and the method for determining the annotated region includes: determining the annotated region based on target tracking;
[0018] The data labeling module is also used to track the object corresponding to the target labeling category, and predict the target labeling area where the object corresponding to the target labeling category appears in each video frame of the video frame sequence, so as to realize the determination of the labeling area based on target tracking.
[0019] In some embodiments of the present application, the data labeling module is also used to track the object corresponding to the target labeling category through multiple target tracking methods, predict multiple appearance areas of the object corresponding to the target labeling category from each video frame of the video frame sequence; and select the target labeling area with the highest confidence from the multiple appearance areas.
[0020] In some embodiments of the present application, the image to be annotated is a video frame sequence, and the method for determining the annotated region includes: determining the annotated region based on motion target detection;
[0021] The data annotation module is also used to perform motion target detection on the video frame sequence to obtain the motion target; when the category corresponding to the motion target is the same as the target annotation category, the image area where the motion target is located is determined as the target annotation area where the object corresponding to the target annotation category is located, so as to realize the determination of the annotation area based on motion target detection.
[0022] In some embodiments of the present application, the data annotation module is also used to subtract each video frame in the video frame sequence and the previous video frame of each video frame to obtain a frame difference image; binarize the frame difference image using a preset threshold to obtain a binarized image, and determine the moving target from the binarized image.
[0023] In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on a target detection model; the data annotation module is also used to perform target detection on the image to be annotated through a target detection model used to detect objects corresponding to the target annotation category, so as to obtain a detection object; wherein the target detection model is trained by annotating image data of objects of the target annotation category; the image area where the detection object is located is determined as the target annotation area where the object corresponding to the target annotation category is located, thereby realizing the determination of the annotation area based on the target detection model.
[0024] In some embodiments of the present application, the data labeling device also includes: a model adjustment module; the model adjustment module is used to use the labeling data to expand the image data of the object labeled with the target labeling category to obtain expanded image data; and use the expanded image data to retrain the target detection model to obtain an optimized target detection model.
[0025] In some embodiments of the present application, the annotation method determination module is also used to determine that the annotation area determination method is to determine the annotation area based on template matching when the image to be annotated contains an object of a geometric shape; when the image to be annotated is a video frame sequence, determine that the annotation area determination method is to determine the annotation area based on target tracking, or to determine the annotation area based on motion target detection; when the image to be annotated contains objects of other shapes besides the geometric shape, determine that the annotation area determination method is to determine the annotation area based on a target detection model.
[0026] In some embodiments of the present application, the data annotation device further includes: an annotation area display module and an annotation area adjustment module;
[0027] The marked area display module is used to mark the target marked area in the image to be marked and display the marked image to be marked;
[0028] The marked area adjustment module is used for, when an adjustment operation on the target marked area is detected, adjusting the target marked area in response to the adjustment operation to obtain an adjusted target marked area;
[0029] The data construction module is further used to construct the annotation data using the adjusted target annotation area and the target annotation category.
[0030] In some embodiments of the present application, the data construction module is further used to capture the image portion corresponding to the target annotation area from the image to be annotated to obtain a captured image; and integrate the target annotation category and the captured image into the annotation data.
[0031] The present application provides a data labeling device, including:
[0032] A memory, for storing executable data annotation instructions;
[0033] The processor is used to implement the data labeling method provided in the embodiment of the present application when executing the executable data labeling instruction stored in the memory.
[0034] An embodiment of the present application provides a computer-readable storage medium storing executable data labeling instructions for causing a processor to execute and implement the data labeling method provided in the embodiment of the present application.
[0035] The embodiment of the present application has the following beneficial effects: the data annotation device first obtains the image to be annotated and the target annotation category, and determines the applicable annotation area determination method for the image to be annotated, and then determines the target annotation area from the image to be annotated based on the annotation area determination method, and finally generates annotation data using the target annotation area and target annotation category of the image to be annotated. In this way, during the data annotation process, the data annotation device can automatically annotate the image to be annotated, so that the annotation data can be quickly obtained, reducing the time required for data annotation, and thus improving the efficiency of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a schematic diagram of the labeled data for the target detection task;
[0037] Figure 2 This is an optional architecture diagram of a data annotation system provided in an embodiment of the present application;
[0038] Figure 3 is a structural schematic diagram of a data labeling device provided in an embodiment of the present application;
[0039] Figure 4 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 1 ;
[0040] Figure 5 is a schematic diagram of specifying target labeling categories provided in an embodiment of the present application;
[0041] Figure 6 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 2 ;
[0042] Figure 7 is a schematic diagram of determining a target annotation area based on template matching provided in an embodiment of the present application;
[0043] Figure 8 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 3 ;
[0044] Fig. 9 It is a schematic diagram of determining a target marking area based on target tracking provided by an embodiment of the present application;
[0045] Fig.10 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 4 ;
[0046] Fig.11 is a schematic diagram of multiple appearance areas of objects corresponding to the target annotation category provided in an embodiment of the present application;
[0047] Fig.12 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 5 ;
[0048] Fig.13 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 6 ;
[0049] Fig.14 is a schematic diagram of determining a moving target by frame difference provided by an embodiment of the present application;
[0050] Fig.15 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 7 ;
[0051] Fig.16 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 8 ;
[0052] Fig.17 is a schematic diagram of a closed-loop lifting annotation effect provided by an embodiment of the present application;
[0053] Fig.18 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 9 ;
[0054] Fig.19is a schematic diagram of an object of a geometric shape provided by an embodiment of the present application;
[0055] Fig. 20 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 ;
[0056] Fig.21 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 one;
[0057] Fig. 22 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 two. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0059] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0061] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0062] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0063] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.
[0064] 2) Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0065] 3) Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.
[0066] 4) Object Detection mainly aims to achieve the following goals: find all objects of interest in the image and determine the location and category of the found objects.
[0067] 5) Template matching refers to searching for a known target in an image.
[0068] 6) Target tracking is to predict the size and position of the target in subsequent video frames when the target size and position are given in the initial frame of the video stream, so that the target can be continuously marked in the video stream to form a visual tracking effect.
[0069] 7) The neural network model is an artificial network system that simulates the animal neural mechanism. It is composed of a large number of simple processing units connected together, has a certain learning ability, and can be applied to multiple fields such as computer vision and natural language processing.
[0070] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, smart homes, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0071] Object detection is an important application direction in artificial intelligence, which aims to find the location and category of the object of interest from the image. At present, the object detection task is mostly implemented by neural network models, but neural network models need a large amount of labeled data when learning to achieve good results.
[0072] The labeled data for target detection is to select the target in the image using the labeled box and give the label of the selected target. For example, Figure 1 This is a schematic diagram of the labeled data for the target detection task. Figure 1 As shown, the labeled data 1-1 selects the targets in each image using a labeled box, and then gives labels for the selected targets, namely, bus 1-11, car 1-12, cat 1-13, chair 1-14, cow 1-15, dining table 1-16, dog 1-17, horse 1-18, motorcycle 1-19 and human 1-110.
[0073] In the related technologies, most of them are done by manually selecting a labeling box in the image and giving a label to the corresponding image part in the box to obtain the labeling data. For example, the image is labeled by professional labelers within the company, or by netizens on the network platform. At present, there are some labeling auxiliary software that can assist manual labeling. For example, it provides a labeling interface corresponding to different labeling tasks to prompt the labeler of the current labeling task, labeling requirements, etc. Although the labeling auxiliary software can give prompts to the labeler when labeling, due to its lack of assistance in the labeling method, even if the labeling auxiliary software is used, the labeler still needs to manually label each picture, and the labeling will take a long time. As a result, the efficiency of data labeling is low and the labor cost is high.
[0074] Furthermore, when manually annotating each image, the size of the annotation box is determined based on the subjective judgment of the annotator. Therefore, the sizes of the annotation boxes of different annotators are likely to be different, making it difficult to ensure the quality of the annotation data.
[0075] The embodiments of the present application provide a data annotation method, apparatus, device, and computer-readable storage medium, which can improve the efficiency of data annotation. The following describes an exemplary application of the data annotation device provided by the embodiments of the present application. The data annotation device provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), etc. It can also be implemented as a server, and can also be implemented as a device cluster consisting of a server and a terminal. Below, an exemplary application of the data annotation device when it is implemented as a server will be described.
[0076] See also Figure 2 , Figure 2 It is an optional architectural diagram of the data annotation system provided in an embodiment of the present application. To support a data annotation application, in the data annotation system 100, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0077] The server 200 is used to obtain the image to be annotated and the target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated; for the image to be annotated, a corresponding annotation area determination method is determined; based on the annotation area determination method, the target annotation area where the object corresponding to the target annotation category is located is determined from the image to be annotated; and annotation data is constructed according to the target annotation area of the image to be annotated and the target annotation category. In this way, the server 200 realizes data annotation.
[0078] Terminal 400 - 1 and terminal 400 - 2 are used to receive the labeled data sent by server 200 through network 300 , so as to use the labeled data for model training.
[0079] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.
[0080] See also Figure 3 , Figure 3 is a structural diagram of a data labeling device provided in an embodiment of the present application, Figure 3 The data annotation device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the data annotation device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not used in the data bus. Figure 3 Various buses are labeled as bus system 540 .
[0081] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0082] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0083] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.
[0084] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.
[0085] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0086] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0087] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB);
[0088] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);
[0089] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.
[0090] In some embodiments, the data annotation device provided in the embodiments of the present application can be implemented in software. Figure 3A data annotation device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, and includes the following software modules: a data acquisition module 5551, an annotation method determination module 5552, a data annotation module 5553, a data construction module 5554, a model adjustment module 5555, an annotation area display module 5556, and an annotation area adjustment module 5557. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented.
[0091] The functions of each module will be described below.
[0092] In other embodiments, the data labeling device provided in the embodiments of the present application can be implemented in hardware. As an example, the data labeling device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data labeling method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.
[0093] Exemplarily, an embodiment of the present application provides a data labeling device, including:
[0094] A memory, for storing executable data annotation instructions;
[0095] The processor is used to implement the data labeling method provided in the embodiment of the present application when executing the executable data labeling instruction stored in the memory.
[0096] Below, the data labeling method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the data labeling device provided in the embodiment of the present application.
[0097] See also Figure 4 , Figure 4 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 1 , will combine Figure 4 The steps shown are explained.
[0098] S101, obtaining an image to be annotated and a target annotation category.
[0099] The embodiment of the present application is implemented in the scenario of generating annotated data with annotation information, that is, generating an annotation box for a specific object in an image to select the specific object and give the category of the specific object. Among them, the specific target can be any object that needs to be annotated, such as animals, pedestrians, objects, vehicles, etc. The annotated data can be used to train network models for different tasks, such as network models for target detection, network models for semantic segmentation, and network models for feature point detection, etc.
[0100] After the data annotation process starts, the data annotation device will first obtain the image waiting for data annotation, that is, the image to be annotated, and the target annotation category. Among them, the target annotation category represents the category of the area where the corresponding object is waiting to be annotated. In other words, in the embodiment of the present application, the data annotation device first clarifies the target annotation category to be annotated when annotating data, and then annotates. When annotating, the data annotation device needs to determine the regional position of the object corresponding to the target annotation category from the image to be annotated.
[0101] It should be noted that the image to be annotated does not refer to a single image, but a general term for all images that need to be annotated. These images that need to be annotated can be a continuous sequence of video frames, or pictures containing objects with regular shapes, or pictures containing objects with irregular shapes. In other words, the images to be annotated can be a sequence of video frames, or various static pictures.
[0102] Furthermore, the image to be annotated may be an image downloaded from the Internet, or an image obtained by photographing a real scene.
[0103] It is understandable that the target annotation category can be specified by the staff. For example, before the start of this round of data annotation, the staff enters the target annotation category through the command line, or determines the image to be annotated as a preset specific category through image recognition based on the acquired image to be annotated.
[0104] For example, Figure 5 is a schematic diagram of specifying target labeling categories provided in an embodiment of the present application. Figure 5In the annotation interface 5-1, an annotation category designation window 5-11 and an image display area 5-12 are provided, wherein the image display area 5-12 is used to display each image in the image data to be annotated. The staff can trigger a switching operation in the annotation category designation window 5-11 (for example, by clicking a switching button with a mouse or pressing an arrow key on a keyboard) to rotate the annotation categories in the annotation category designation window 5-11. When the staff stops the switching operation, the data annotation device obtains the content displayed in the annotation category designation window 5-11 and obtains the target annotation category. For example, the label displayed in the annotation category designation window 5-11 is obtained: chess piece 5-111 (pawn), and the chess piece 5-111 is used as the target annotation category.
[0105] The target annotation category can also be specified by the data annotation device. For example, the data annotation device randomly selects a annotation category from the preset annotation categories as the target annotation category, or generates a similar annotation category based on the preset annotation category as the target annotation category (for example, generating Go based on chess, etc.).
[0106] In the embodiment of the present application, the target annotation category can be various categories such as cats, pedestrians, plants, etc.
[0107] S102: for the image to be annotated, determine a corresponding annotation region determination method.
[0108] The data labeling device analyzes the image to be labeled to clarify the characteristics of each image contained in the image to be labeled, for example, whether it is a continuous video frame sequence, whether there are objects with regular shapes in the image content, and then determines the appropriate labeling area determination method for the image to be labeled based on the characteristics of each image contained in the image to be labeled.
[0109] It should be noted that the method for determining the annotation area refers to a method for determining the annotation area from the image to be annotated, so that in the embodiment of the present application, the method for determining the annotation area determined by the data annotation device can be adapted to the characteristics of the image to be annotated, so that through the method for determining the annotation area, the target annotation area corresponding to the target annotation category can be quickly determined for the image to be annotated. For example, when the image to be annotated is a video frame sequence, the data annotation device can obtain the annotation area determination method based on a target tracking method that can process the video, or a motion target detection method; when the image to be annotated is a static image, the data annotation device can obtain the annotation area determination method based on a target detection model that can process static images, or template matching.
[0110] It is understandable that the video frames in the video frame sequence are essentially static images, so the method of determining the annotation area corresponding to the static image by the data annotation device can also be applied to the video frame sequence.
[0111] In some embodiments, the data annotation device may input the image to be annotated into a trained calibration region determination model, so as to analyze the characteristics of the image to be annotated through the calibration region determination model, thereby determining the calibration region determination method. It is understandable that the calibration region determination model is trained using images with appropriate calibration region determination method labels.
[0112] In other embodiments, the data annotation device may also match the content of the image to be annotated with the example images corresponding to each preset region determination method. When the image to be annotated matches the example image corresponding to a preset region determination method, the data annotation device will use the preset region determination method as the annotation region determination method.
[0113] Generally speaking, the image to be annotated may include a sequence of video frames, pictures containing objects of regular shapes, pictures containing objects of irregular shapes, and the like. Therefore, the method for determining the annotated area in the embodiments of the present application may be to determine the annotated area based on template matching, to determine the annotated area based on target tracking of the video, to determine the annotated area based on motion target detection of the video, to determine the annotated area based on target detection, and the like.
[0114] S103 : Based on the marking region determination method, determine the target marking region where the object corresponding to the target marking category is located from the image to be marked.
[0115] After determining the annotation region determination method, the data annotation device uses the annotation region determination method to determine the annotation region containing the object corresponding to the target annotation category from the image to be annotated, and uses the annotation region as the target annotation region. That is, in the embodiment of the present application, the target annotation region contains the object corresponding to the target annotation category.
[0116] It should be noted that the method for determining the annotation area of the image to be annotated is determined based on the characteristics of the image to be annotated. Therefore, determining the target annotation area from the image to be annotated based on the annotation area determination method is essentially determining the target annotation area based on the characteristics of the image to be annotated itself.
[0117] It is understandable that the target annotation area refers to the general term for the annotation areas corresponding to the images contained in the to-be-annotated image, and does not specifically refer to a certain annotation area. The target annotation area can be the vertex coordinates of the rectangular area where the object corresponding to the target annotation category is located, or the center point and radius of the circular area where the object corresponding to the target annotation category is located, etc., which is not limited in the embodiments of the present application.
[0118] S104: constructing annotation data according to the target annotation area and the target annotation category of the image to be annotated.
[0119] The data labeling device integrates the target labeling area and target labeling category into the image to be labeled, and the resulting data is the labeled data. In this way, the data labeling device can automatically label the image to be labeled without the need for manual operation by the labeler, greatly reducing the time and labor cost required for data labeling.
[0120] In some embodiments, the data annotation device may annotate the target annotation region on the image to be annotated to obtain the image to be annotated after the region is annotated, and then use the target annotation category as a label of the image to be annotated after the region is annotated to obtain the annotation data.
[0121] In other embodiments, the data annotation device can use the target annotation area and the target annotation category as labels of the image to be annotated, thereby obtaining annotation data. In this case, it is not necessary to annotate the target annotation area in the image to be annotated, and the obtained annotation data can also be used for training classification models, etc.
[0122] In other embodiments, the data labeling device may also extract the image portion corresponding to the target labeling area from the image to be labeled, and then use the extracted image portion as a new image, and then use the target labeling category as a label of the new image to obtain the labeling data.
[0123] In the embodiment of the present application, the data annotation device will first obtain the image to be annotated and the target annotation category, and determine the applicable annotation area determination method for the image to be annotated, and then determine the target annotation area from the image to be annotated based on the annotation area determination method, and finally generate annotation data using the target annotation area and target annotation category of the image to be annotated. In this way, during the data annotation process, the data annotation device can automatically annotate the image to be annotated, so that the annotation data can be quickly obtained, reducing the time required for data annotation, and thus improving the efficiency of data annotation. In addition, by automatically annotating the image to be annotated through the data annotation device, there is no need for the annotation personnel to perform manual operations during data annotation, which reduces the labor cost of data annotation.
[0124] based on Figure 4 , see Figure 6 , Figure 6 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on template matching. At this time, based on the method for determining the annotation area, the target annotation area where the object corresponding to the target annotation category is located is determined from the image to be annotated, that is, the specific implementation process of S103 may include: S1031-S1032, as follows:
[0125] S1031 . Match the acquired template object corresponding to the target annotation category to the candidate object from the image to be annotated.
[0126] When it is determined that the method for determining the annotation area for the image to be annotated is to determine the annotation area based on template matching, the data annotation device will obtain the template object corresponding to the target annotation category, and then use the obtained template object to perform template matching on the image to be annotated to determine the candidate object that matches the template object from the image to be annotated.
[0127] In the embodiment of the present application, the template objects corresponding to the target annotation category may include objects of regular shapes, such as chess pieces, traffic signs, etc. The template objects corresponding to the target annotation category may also include objects with obvious identification, such as pets wearing clothes of a specific style, flags of a specific color, etc. Of course, the template objects of the target annotation category may also be other objects, which are not limited in the present application.
[0128] It is understandable that when the data annotation device performs template matching on the template object of the target annotation category and the image to be annotated, it essentially searches for image content similar to the template object of the target annotation category in the image to be annotated, and the obtained image content is the candidate object.
[0129] Furthermore, the data annotation device first translates the template object on the image to be annotated, calculates the similarity between the image content covered by the template object and the template object, and when the similarity reaches a certain level, the image content covered by the template object is determined as a candidate object.
[0130] S1032: Determine the image region where the candidate object is located as the target annotation region, so as to determine the annotation region based on template matching.
[0131] After obtaining the candidate object, the data labeling device will first determine the image area where the candidate object is located in the image to be labeled, and use the determined image area as the target labeling area. In this way, the data labeling device can determine the labeling area based on template matching.
[0132] It is understandable that when multiple candidate objects are determined from the image to be annotated, the data annotation device determines the image area where each candidate object is located as the target annotation area, so that there are multiple target annotation areas in one image. Furthermore, the data annotation device can also intercept and process each candidate object as a separate image, so that multiple images with target annotation areas can be obtained from one image of the image to be annotated (each image has one target annotation area).
[0133] For example, Figure 7 is a schematic diagram of determining a target annotation area based on template matching provided in an embodiment of the present application. Figure 7 , the object in the template object 7-1 is a chess piece (pawn), and the image 7-2 is the image to be annotated. The data annotation device performs template matching on the image 7-2 based on the template object 7-1, and matches 5 chess pieces from it. These chess pieces are candidate objects. Finally, the data annotation device determines the image area where the candidate object is located as the target annotation area 7-3, thus realizing the process of determining the annotation area based on template matching.
[0134] In an embodiment of the present application, the data labeling device first performs template matching on the template object of the acquired target labeling category object and the image to be labeled to determine a matching candidate object from the image to be labeled, and then determines the image area where the candidate object is located as the target labeling area to implement the process of determining the labeling area based on template matching, thereby reducing the time required to determine the target labeling area.
[0135] based on Figure 4 , see Figure 8 , Figure 8 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, the image to be annotated is a sequence of video frames, and the method for determining the annotated region includes: determining the annotated region based on target tracking, thereby, based on the annotated region determination method, determining the target annotated region where the object corresponding to the target annotated category is located from the image to be annotated, that is, the specific implementation process of S103 may include: S1033, as follows:
[0136] S1033 , by tracking the object corresponding to the target annotation category, predicting the target annotation area where the object corresponding to the target annotation category appears in each video frame of the video frame sequence, so as to determine the annotation area based on target tracking.
[0137] In a video frame sequence, different forms and positions of the same moving object can usually be reflected. Therefore, if data is annotated for each video frame in the video frame sequence, the generalization of the annotated data can be greatly improved. Target tracking and motion target detection are both methods that can determine moving objects from a continuous video frame sequence. Therefore, when the image to be annotated is a video frame sequence, the method for determining the annotation area determined by the data annotation device for the image to be annotated can be based on target tracking to determine the annotation area. At this time, the data annotation device can first track the object corresponding to the target annotation category in the video frame sequence, thereby determining the area where the object appears in each video frame and obtaining the target annotation area.
[0138] It can be understood that in some embodiments, the object corresponding to the target annotation category can be specified by the annotator. For example, the annotator selects an object in the initial video frame of the video frame sequence, and the data annotation device will use the object as the object corresponding to the target annotation category.
[0139] In other embodiments, the object corresponding to the target annotation category can be determined by the data annotation device itself. For example, the data annotation device can match the preset object stored for the target annotation category with the initial video frame in the video frame sequence to determine from the initial video frame an object that has a certain degree of similarity with the preset object as an object of the target annotation category for tracking.
[0140] The data annotation device determines the object appearance area corresponding to the target annotation category in each video frame of the video frame sequence through target tracking, and determines the appearance area of the object corresponding to the target annotation category in each video frame as the target annotation area. In this way, the data annotation device can determine the annotation area based on target tracking and determine the annotation area for a large number of images at one time.
[0141] For example, Fig. 9 Schematic diagram of determining target annotation areas based on target tracking provided by an embodiment of the present application. Tracking object 9-1 is a shopping bag, and video frame 9-2, video frame 9-3, and video frame 9-4 are three video frames in a video frame sequence containing a shopping bag. The data annotation device performs target tracking on object 9-1 of the target annotation category, so that in each video frame of the video frame sequence, the appearance area 9-5 of the tracking object 9-1, i.e., the shopping bag, is determined, and these appearance areas 9-5 are used as target annotation areas to implement the process of determining the annotation area based on target tracking.
[0142] In the embodiment of the present application, the data annotation device tracks the object corresponding to the target annotation category in the video frame sequence, thereby determining the target annotation area of the object corresponding to the target annotation category in each video frame. In this way, the annotation area can be determined for a large number of video frames in the video frame sequence at one time, thereby improving the efficiency of data annotation.
[0143] See also Fig.10 , Fig.10 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 4 In some embodiments of the present application, by tracking the object corresponding to the target annotation category, predicting the target annotation area where the object corresponding to the target annotation category appears from each video frame in the video frame sequence, that is, the specific implementation process of S1033, may include: S1033a-S1033b, as follows:
[0144] S1033a, tracking the object corresponding to the target annotation category by using multiple target tracking methods, and predicting multiple appearance areas of the object corresponding to the target annotation category from each video frame in the video frame sequence.
[0145] The data annotation device tracks the object corresponding to the target annotation category in the video frame sequence through a variety of different target tracking methods. Through each target tracking method, an appearance area of the object can be predicted from each video frame in the video frame sequence. Thus, the data annotation device tracks the object through a variety of target tracking methods, and can predict multiple appearance areas of the object corresponding to the target annotation category in each video frame in the video frame sequence.
[0146] It can be understood that the various target tracking methods may include a single target tracking algorithm (SiamRP N), a continuous domain convolution operation tracking algorithm (CCOT), etc., and may also include a tracking algorithm based on a fully convolutional twin network (SiamFC), etc.
[0147] For example, Fig.11 is a schematic diagram of multiple appearance areas of objects corresponding to target annotation categories provided in an embodiment of the present application. Fig.11, video frame 11-11, video frame 11-12 and video frame 11-13 are three video frames in video frame sequence 11-1, and the object corresponding to the target annotation category in video frame sequence 11-1 is shopping bag 11-A; video frame 11-21, video frame 11-22 and video frame 11-23 are three video frames in video frame sequence 11-2, and the object corresponding to the target annotation category in video frame sequence 11-2 is cat 11-B; video frame 11-31, video frame 11-32 and video frame 11-33 are three video frames in video frame sequence 11-3, and the object corresponding to the target annotation category in video frame sequence 11-3 is octopus 11-C. The data annotation device uses three different target tracking methods, namely target tracking method 11-4, target tracking method 11-5 and target tracking method 11-6, to track the shopping bag 11-A, cat 11-B and octopus 11-C respectively, so that three different appearance areas can be predicted in each video frame of the video frame sequence 11-1, the video frame sequence 11-2 and the video frame sequence 11-3.
[0148] It should be noted that, while the data annotation device predicts multiple appearance areas of the object corresponding to the target annotation category in each video frame, it also obtains the confidence of each appearance area, thereby obtaining multiple confidences corresponding to the multiple appearance areas one by one.
[0149] S1033b. Select a target annotation area with the highest confidence from multiple appearance areas.
[0150] After predicting multiple appearance areas of the tracked object in each video frame, the data annotation device will sort the confidence levels corresponding to the multiple appearance areas, and determine the appearance area with the highest confidence level as the target annotation area.
[0151] It should be noted that, in some other embodiments, the data annotation device may also weight the coordinates of multiple appearance areas to obtain weighted appearance areas, and determine the weighted appearance areas as target annotation areas.
[0152] In the embodiment of the present application, the data annotation device can predict multiple appearance regions in each video frame of the video frame sequence for the object corresponding to the target annotation category, and select the target annotation region with the highest confidence from the multiple appearance regions. In this way, the target annotation region determined by the data annotation device will be sufficiently accurate, so that the quality of the annotation data is high.
[0153] based on Figure 4 , see Fig.12 , Fig.12 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 5In some embodiments of the present application, the image to be annotated is a sequence of video frames, and the method for determining the annotated region includes: determining the annotated region based on motion target detection. In this case, based on the annotated region determination method, the target annotated region where the object corresponding to the target annotated category is located is determined from the image to be annotated, that is, the specific implementation process of S103 may include: S1034-S1035, as follows:
[0154] S1034: Perform moving target detection on the video frame sequence to obtain a moving target.
[0155] When the image to be annotated is a video frame sequence, and the method for determining the annotation area for the image to be annotated is to determine the annotation area based on running target detection, the data annotation device will start the process of motion target detection for the video frame sequence, thereby determining the moving target in the data annotation device.
[0156] It is understandable that the data annotation device can detect moving targets in video frame sequences through methods such as frame difference method, optical flow tracking method, background modeling method, etc. to obtain moving targets.
[0157] S1035. When the category corresponding to the moving target is the same as the target annotation category, determine the image region where the moving target is located as the target annotation region where the object corresponding to the target annotation category is located, so as to determine the annotation region based on moving target detection.
[0158] There may be more than one moving target in the video frame sequence, and these moving targets may not belong to the target annotation category. Therefore, the data annotation device will compare the category to which the moving target belongs with the target annotation category. When the category corresponding to the moving target is the same as the target annotation category, the data annotation device will determine the image area where the moving target is located from each video frame in the video frame sequence, and use the image area as the target annotation area. In this way, the data annotation device completes the process of determining the annotation area based on moving target detection.
[0159] It should be noted that the category to which the moving target belongs can be obtained by randomly extracting a frame from the video frame sequence by the data annotation device and classifying and identifying the extracted video frame using the classification model. The category to which the moving target belongs can also be specified by the annotation personnel after the data annotation device provides the video frame sequence to the annotation personnel.
[0160] It can be understood that when the data annotation device determines the image area where the moving target is located from each video frame, the pixel coordinates of the upper left corner and the lower right corner of the moving target can be used to jointly determine a rectangular image area, or the pixel coordinates of the center position of the moving target and the distance between the pixels farthest from the center position can be used as the radius to jointly determine a circular image area, etc.
[0161] In the embodiment of the present application, the data annotation device first performs moving target detection on the video frame sequence, and then compares the category of the detected moving target with the target annotation category. When the category of the moving target is consistent with the target annotation category, the data annotation device will use the image area where the moving target is located as the target annotation area. In this way, the data annotation device can automatically determine the target annotation area through moving target detection.
[0162] See also Fig.13 , Fig.13 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 6 In some embodiments of the present application, the moving target is detected on the video frame sequence to obtain the moving target, that is, the specific implementation process of S1034 may include: S1034a-S1034b, as follows:
[0163] S1034a, performing subtraction on each video frame in the video frame sequence and the previous video frame of each video frame to obtain a frame difference image.
[0164] S1034b, binarizing the frame difference image using a preset threshold to obtain a binarized image, and determining a moving target from the binarized image.
[0165] When the data annotation device detects moving targets in a video frame sequence, it will first perform a difference between each video frame in the video frame sequence and the previous video frame of each video frame to determine the frame difference between each video frame and its previous video frame. In this way, the moving object in each video frame can be detected, thereby obtaining the moving target. After calculating the frame difference, the data annotation device can also use a preset threshold to binarize the frame difference image, that is, compare each pixel in the frame difference image with the preset threshold, set the pixel value to 255 if it is greater than the threshold, and set the pixel value to 0 if it is less than the threshold, to obtain a binary image. The white target in the binary image is the moving target in the foreground.
[0166] For example, Fig.14 This is a schematic diagram of determining a moving target by frame difference provided by an embodiment of the present application. The data annotation device subtracts each video frame 14-1 in the video frame sequence from the previous video frame 14-2 of each video frame to obtain a frame difference map, and then binarizes the frame difference map using a set threshold 14-3 to obtain a binarized image 14-4. The white portion in the binarized image 14-4 is the determined moving target 14-5.
[0167] In the embodiment of the present application, the data annotation device can first obtain a frame difference image between each video frame in the video frame sequence and its corresponding previous video frame by difference, then binarize the frame difference image by a preset threshold, and finally determine the moving target from the binarized image. In this way, the data annotation device can automatically determine the moving target in the video frame sequence.
[0168] based on Figure 4 , see Fig.15 , Fig.15 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 7 In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on the target detection model, thereby determining the target annotation area where the object corresponding to the target annotation category is located from the image to be annotated based on the annotation area determination method, that is, the specific implementation process of S103 may include: S1036-S1037, as follows:
[0169] S1036. Perform target detection on the image to be annotated using a target detection model for detecting objects corresponding to the target annotated category to obtain a detected object.
[0170] When the data labeling device determines the labeling area based on the target detection model, it will first obtain the target detection model corresponding to the target labeling category, and then input the image to be labeled into the target detection model to detect the detection object from the image to be labeled.
[0171] It should be noted that the target detection model is trained by annotating the image data of the target labeled category. These images are manually annotated by the annotators and obtained through data augmentation.
[0172] In some embodiments, in order to solve the problem of overfitting, the data labeling device will first use images labeled with objects of various categories to pre-train an initial detection model, and then use images labeled with objects corresponding to the target labeling category to train the initial detection model to obtain a target detection model.
[0173] S1037: Determine the image region where the detected object is located as the target annotation region where the object corresponding to the target annotation category is located, so as to determine the annotation region based on the target detection model.
[0174] After determining the detection object, the data annotation device will determine the image area where the detection image is located in the image to be annotated, and use the image area as the target annotation area.
[0175] In the embodiment of the present application, the data annotation device first detects the detection object from the image to be annotated using the target detection model corresponding to the target annotation category, and the area where the detection object is located is determined as the target annotation area. In this way, the data annotation device automatically determines the target annotation area through the target detection model, reducing the time required to determine the target annotation area.
[0176] based on Figure 4 , see Fig.16 , Fig.16 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 8 In some embodiments of the present application, when the method for determining the annotation area includes determining the annotation area based on the target detection model, the data annotation device constructs the annotation data according to the target annotation area of the image to be annotated and the target annotation category, that is, after S104, the method may further include: S105-S106, as follows:
[0177] S105 , using the annotation data, expanding the image data of the object corresponding to the target annotation category to obtain expanded image data.
[0178] S106. Retrain the target detection model using the expanded image data to obtain an optimized target detection model.
[0179] After the data annotation device determines the target annotation area from the image to be annotated based on the target detection model and constructs the annotation data according to the target annotation area and the target annotation category, the data annotation device adds the annotation data to the image data of the object corresponding to the annotated target annotation category to expand the image data and obtain the expanded image data. Next, the data annotation device will use the expanded image data to train the target detection model. Through this iteration, the learning effect of the target detection model can be improved, and an optimized target detection model can be obtained. In the future, the optimized target detection model can be used to obtain a more accurate target annotation area, that is, a closed loop is used to improve the annotation effect.
[0180] For example, Fig.17 Schematic diagram of the closed-loop lifting labeling effect provided by the embodiment of the present application. Fig.17As shown, the data annotation device first uses the image data 17-1 of the object with the target annotation category to train 17-2 the target detection model 17-3, and then uses the target detection model 17-3 to annotate the data to be annotated 17-4 to obtain the annotated data 17-5, and then adds 17-6 the annotated data 17-5 to the image data 17-1 to perform a new round of model training. With the continuous expansion of image data, the target annotated area identified by the target detection model will become more and more accurate, so that the annotation effect will become better and better.
[0181] In an embodiment of the present application, the data labeling device can use the labeled data to expand the image data, and use the expanded image data to retrain the target detection model, so that the learning effect of the target detection model is better, and an optimized target detection model is obtained, so that the subsequent labeling effect is better.
[0182] based on Figure 4 , see Fig.18 , Fig.18 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 9 In some embodiments of the present application, for the image to be annotated, determining the corresponding annotation region determination method, that is, the specific implementation process of S102, may include: any one of steps S1021-S1023, as follows:
[0183] S1021 . When the image to be annotated contains an object of geometric shape, determine the annotated region by using template matching as the basis.
[0184] When the image to be annotated includes an object of geometric shape, it means that the image to be annotated has obvious features and is suitable for template matching. Therefore, the data annotation device will use template matching as a method for determining the annotation area.
[0185] It should be noted that the geometric shape may include common shapes such as circle, rectangle, triangle, etc., and may also include shapes obtained by combining common shapes such as circle, triangle, rectangle, etc.
[0186] For example, Fig.19 Schematic diagram of an object of geometric shape provided in an embodiment of the present application. Fig.19 As shown, there is an obvious outer ring 19-2 in the chess piece 19-1 (pawn) contained in a certain image, and the fixed-size ring in the image can be found by template matching. Therefore, the data labeling device will determine the labeling area of the image based on template matching.
[0187] S1022: When the image to be annotated is a video frame sequence, a method for determining the annotated region is to determine the annotated region based on target tracking or to determine the annotated region based on moving target detection.
[0188] The video frame sequence has a certain continuity in image content, and both target tracking and motion target detection use the continuity of the video frame sequence to determine the target. Therefore, when the image to be annotated is a video frame sequence, the data annotation device will select one of the methods of determining the annotation area based on target tracking or based on motion target detection as the annotation area determination method.
[0189] S1023: When the image to be annotated contains objects of shapes other than geometric shapes, a method for determining the annotated region is to determine the annotated region based on an object detection model.
[0190] The target detection model is capable of detecting a variety of irregular targets. Therefore, when the image to be annotated contains objects of shapes other than geometric shapes, the data annotation device will determine the annotation area determination method as determining the annotation area based on target detection.
[0191] In an embodiment of the present application, the data annotation device analyzes the image to be annotated and determines the characteristics of the image to be annotated, thereby determining a corresponding annotation area determination method for the image to be annotated, so as to facilitate subsequent determination of the target annotation area from the image to be annotated based on the annotation area determination method.
[0192] based on Figure 4 , see Fig. 20 , Fig. 20 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 In some embodiments of the present application, after determining the target annotation area where the object corresponding to the target annotation category is located from the image to be annotated based on the annotation area determination method, before constructing the annotation data according to the target annotation area of the image to be annotated and the target annotation category, that is, after S103 and before S104, the method may further include: S107-S108, as follows:
[0193] S107 , marking the target marking area in the image to be marked, and displaying the marked image to be marked.
[0194] The data annotation device marks the target annotation area from the image to be annotated, obtains the annotated image to be annotated, and displays the image to be annotated on the image interface so that the annotator can check whether the target annotation area is correct.
[0195] S108 : When an adjustment operation on the target marked area is detected, the target marked area is adjusted in response to the adjustment operation to obtain an adjusted target marked area.
[0196] When the annotator finds that the target annotation area is wrong, the target annotation area can be adjusted. When the data annotation device detects the adjustment operation of the annotator, it will adjust the target annotation area in response to the adjustment operation so that the adjusted target annotation area can accurately mark the object to be marked in the image to be marked.
[0197] It is understandable that the adjustment operation may be a fine adjustment of the size of the target annotation area, for example, enlarging or reducing the target annotation size, etc. The adjustment operation may also be an adjustment of the position of the target annotation area, for example, moving the target annotation area, etc.
[0198] After the data annotation device obtains the adjusted target annotation area, based on Figure 4 , see Fig.21 , Fig.21 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 1. Constructing annotation data according to the target annotation area of the image to be annotated and the target annotation category, that is, the specific implementation process of S104, may include: S1041, as follows:
[0199] S1041. Construct annotation data using the adjusted target annotation area and target annotation category.
[0200] After obtaining the adjusted target annotation area, the data annotation device will integrate the adjusted target annotation area and the target annotation category to obtain more accurate annotation data.
[0201] In an embodiment of the present application, the data labeling device can also label the target labeling area in the image to be labeled, display it to the labeling personnel, and adjust the target labeling area in response to the labeling personnel's adjustment operation, so as to construct the labeling data using the adjusted target labeling area and target labeling category to improve the accuracy of the labeling data.
[0202] based on Figure 4 , see Fig. 22 , Fig. 22 This is an optional process diagram of the data annotation method provided in the embodiment of the present application. Figure 10 2. In some embodiments of the present application, the annotation data is constructed according to the target annotation area of the image to be annotated and the target annotation category, that is, the specific implementation process of S104 may include: S1042-S1043, as follows:
[0203] S1042: From the image to be annotated, cut out the image portion corresponding to the target annotated area to obtain a cutout image.
[0204] S1043: Integrate the target label category and the captured image into label data.
[0205] When constructing annotation data, the data calibration device will first extract the image part corresponding to the target annotation area from the image to be annotated to obtain a screenshot image, and then integrate the target annotation category into the label of the screenshot image to obtain annotation data that does not contain other irrelevant objects.
[0206] In an embodiment of the present application, the data labeling device will capture a screenshot image corresponding to the target labeling area from the image to be labeled, and integrate the target labeling category and the captured image to obtain labeling data. The labeling data obtained in this way will not contain objects that are not related to the target labeling category.
[0207] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.
[0208] The embodiment of the present application is implemented in a scenario where data annotation is performed on an image (image to be annotated) with the aid of a computer (data annotation device).
[0209] First, the computer will use appropriate annotation methods to distinguish the differences in the images. The annotated objects contained in the image are generally divided into three types: characteristic objects, objects in the video stream, and general objects.
[0210] For characteristic targets (geometric objects), the computer will use template matching technology (determine the annotation area based on template matching) to perform annotation. For example, for a chess piece with an outer ring, the computer can use a fixed-size ring in the image to locate the chess piece, so as to automatically generate annotation boxes (target annotation areas) for multiple chess pieces of the same type from a picture containing multiple chess pieces. When saving the annotation data, the annotator only needs to press the save button to generate multiple images of the same type of chess piece at one time (from the image to be annotated, the image corresponding to the target annotation area is intercepted to obtain the intercepted image).
[0211] The target in the video stream (video frame sequence) is generally a moving target such as a pedestrian or a vehicle. The computer can use target tracking (determine the annotation area based on target tracking) and motion target detection (determine the annotation area based on motion target detection). The computer can use SiamRPN, CCOT, SiamFC, etc. for target tracking, that is, the object marked in the first frame of the video stream (the tracking object specified by the target annotation category) is continuously tracked and the tracking frame of each frame is obtained. If the tracking frame is correct, the computer uses the tracking frame as the annotation frame (target annotation area); if the tracking frame is incorrect, the operation of the annotator (adjustment operation), such as re-selecting the object or fine-tuning the tracking frame, can be used to obtain a new annotation frame (adjusted target annotation frame), thereby obtaining the annotation data. Of course, after obtaining the new annotation frame, the computer will update the target tracker with the new annotation frame to perform more accurate target tracking in the future.
[0212] Commonly used methods for moving target detection include frame difference method and optical flow method, both of which generate target detection frames based on the changes between the upper and lower frames (frame difference images). For example, the current frame (each video frame) and the previous frame (the previous video frame of each video frame) are subtracted to obtain the contour of the moving target and finally generate a labeling frame.
[0213] For general targets (objects of shapes other than geometric shapes), although there is no specific method to assist in generating annotation boxes, the computer can perform data augmentation on a small number of images manually annotated by the annotators to generate a data set (image data of objects with target annotation categories annotated), and then use the data set to train the pre-trained model to obtain a neural network detection model (target detection model) for targets of this category, and use the neural network detection model to perform target detection on the image to obtain an annotation box (detect the object, and determine the image area where the detected object is located as the target annotation area). This method can assist in generating relatively correct annotation boxes, and the annotators only need to fine-tune the annotation boxes or select the missed annotation boxes. In addition, the computer can use images with annotation boxes to expand the data set (use the annotation data to expand the image data of objects with target annotation categories annotated), and then use the expanded data set (expanded image data) to iteratively train the neural network detection model to improve the learning effect of the neural network detection model and further improve the annotation efficiency.
[0214] From the above, it can be seen that the computer can automatically classify the targets in the image, thereby determining the appropriate annotation method for the image, and then use the corresponding annotation method to determine the annotation box for the image. In this way, there is no need for the annotator to manually select the annotation box for the image, which reduces the generation time of the annotation box, improves the generation efficiency of the annotation box, improves the quality of the annotation data, and reduces the labor cost of annotation.
[0215] The following is a description of an exemplary structure of the data annotation device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the data annotation device 555 of the memory 550 may include:
[0216] The data acquisition module 5551 is used to acquire the image to be annotated and the target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated;
[0217] The marking method determination module 5552 is used to determine a corresponding marking area determination method for the image to be marked;
[0218] A data annotation module 5553 is used to determine, based on the annotation area determination method, a target annotation area where an object corresponding to the target annotation category is located from the image to be annotated;
[0219] The data construction module 5554 is used to construct the annotation data according to the target annotation area of the image to be annotated and the target annotation category.
[0220] In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on template matching; the data annotation module 5553 is also used to match the candidate object from the image to be annotated for the template object corresponding to the target annotation category obtained; and determining the image area where the candidate object is located as the target annotation area to achieve the determination of the annotation area based on template matching.
[0221] In some embodiments of the present application, the image to be annotated is a video frame sequence, and the method for determining the annotated region includes: determining the annotated region based on target tracking;
[0222] The data labeling module 5553 is also used to track the object corresponding to the target labeling category, and predict the target labeling area where the object corresponding to the target labeling category appears in each video frame of the video frame sequence, so as to realize the determination of the labeling area based on target tracking.
[0223] In some embodiments of the present application, the data labeling module 5553 is also used to track the object corresponding to the target labeling category through multiple target tracking methods, and predict multiple appearance areas of the object corresponding to the target labeling category from each video frame of the video frame sequence; from the multiple appearance areas, select the target labeling area with the highest confidence.
[0224] In some embodiments of the present application, the image to be annotated is a video frame sequence, and the method for determining the annotated region includes: determining the annotated region based on motion target detection;
[0225] The data labeling module 5553 is also used to perform motion target detection on the video frame sequence to obtain the motion target; when the category corresponding to the motion target is the same as the target labeling category, the image area where the motion target is located is determined as the target labeling area where the object corresponding to the target labeling category is located, so as to realize the determination of the labeling area based on motion target detection.
[0226] In some embodiments of the present application, the data annotation module 5553 is also used to subtract each video frame in the video frame sequence and the previous video frame of each video frame to obtain a frame difference image; binarize the frame difference image using a preset threshold to obtain a binarized image, and determine the moving target from the binarized image.
[0227] In some embodiments of the present application, the method for determining the annotation area includes: determining the annotation area based on a target detection model; the data annotation module 5553 is also used to perform target detection on the image to be annotated through a target detection model used to detect objects corresponding to the target annotation category, so as to obtain a detection object; wherein the target detection model is trained by annotating image data of objects of the target annotation category; determining the image area where the detection object is located as the target annotation area where the object corresponding to the target annotation category is located, thereby realizing the determination of the annotation area based on the target detection model.
[0228] In some embodiments of the present application, the data labeling device 555 also includes: a model adjustment module 5555; the model adjustment module 5555 is used to use the labeling data to expand the image data of the object labeled with the target labeling category to obtain expanded image data; use the expanded image data to retrain the target detection model to obtain an optimized target detection model.
[0229] In some embodiments of the present application, the annotation method determination module 5552 is also used to determine that the annotation area determination method is to determine the annotation area based on template matching when the image to be annotated contains an object of a geometric shape; when the image to be annotated is a video frame sequence, determine that the annotation area determination method is to determine the annotation area based on target tracking, or to determine the annotation area based on motion target detection; when the image to be annotated contains objects of other shapes besides the geometric shape, determine that the annotation area determination method is to determine the annotation area based on a target detection model.
[0230] In some embodiments of the present application, the data annotation device 555 further includes: an annotation area display module 5556 and an annotation area adjustment module 5557;
[0231] The marked area display module 5556 is used to mark the target marked area in the image to be marked and display the marked image to be marked;
[0232] The annotation region adjustment module 5557 is configured to adjust the target annotation region in response to an adjustment operation on the target annotation region when an adjustment operation on the target annotation region is detected, so as to obtain an adjusted target annotation region;
[0233] The data construction module 5554 is further used to construct the annotation data using the adjusted target annotation area and the target annotation category.
[0234] In some embodiments of the present application, the data construction module 5554 is further used to capture the image portion corresponding to the target annotation area from the image to be annotated to obtain a captured image; and integrate the target annotation category and the captured image into the annotation data.
[0235] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data labeling method described above in the embodiment of the present application.
[0236] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein executable data labeling instructions are stored. When the executable data labeling instructions are executed by a processor, the processor will be caused to execute the data labeling method provided by the present application embodiment, for example, Figure 4 The method shown.
[0237] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0238] In some embodiments, the executable data annotation instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0239] As an example, the executable data annotation instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0240] As an example, the executable data annotation instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.
[0241] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A data labeling method, characterized in that: include: Obtaining an image to be annotated and a target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated; When the image to be annotated contains an object of geometric shape, the method of determining the annotated area is to determine the annotated area based on template matching; When the image to be annotated is a video frame sequence, determining the annotated region determination method is to determine the annotated region based on target tracking, or to determine the annotated region based on moving target detection; When the image to be annotated contains objects of other shapes except the geometric shape, determining the annotation area determination method is to determine the annotation area based on a target detection model; Based on the annotation area determination method, determining a target annotation area where an object corresponding to the target annotation category is located from the image to be annotated; The annotation data is constructed according to the target annotation area of the image to be annotated and the target annotation category.
2. The method according to claim 1, characterized in that: The method for determining the annotation area includes: determining the annotation area based on template matching; determining the target annotation area where the object corresponding to the target annotation category is located from the image to be annotated based on the method for determining the annotation area includes: Matching the acquired template object corresponding to the target annotation category to the candidate object from the image to be annotated; The image region where the candidate object is located is determined as the target annotated region, so as to implement the determination of the annotated region based on template matching.
3. The method according to claim 1, characterized in that The image to be annotated is a sequence of video frames, and the method for determining the annotated area includes: determining the annotated area based on target tracking; The step of determining the target annotation region where the object corresponding to the target annotation category is located from the image to be annotated based on the annotation region determination method includes: By tracking the object corresponding to the target annotation category, the target annotation area where the object corresponding to the target annotation category appears is predicted from each video frame of the video frame sequence, so as to achieve the determination of the annotation area based on target tracking.
4. The method according to claim 3, characterized in that The method of tracking the object corresponding to the target annotation category and predicting the target annotation area where the object corresponding to the target annotation category appears from each video frame of the video frame sequence includes: Tracking the object corresponding to the target label category by multiple target tracking methods, and predicting multiple appearance areas of the object corresponding to the target label category from each video frame of the video frame sequence; The target marked area with the highest confidence is selected from the multiple appearance areas.
5. The method according to claim 1, characterized in that The image to be annotated is a sequence of video frames, and the method for determining the annotated area includes: determining the annotated area based on moving target detection; The step of determining the target annotation region where the object corresponding to the target annotation category is located from the image to be annotated based on the annotation region determination method includes: Performing moving target detection on the video frame sequence to obtain a moving target; When the category corresponding to the moving target is the same as the target annotation category, the image area where the moving target is located is determined as the target annotation area where the object corresponding to the target annotation category is located, so as to achieve the determination of the annotation area based on moving target detection.
6. The method according to claim 5, characterized in that The performing moving target detection on the video frame sequence to obtain the moving target includes: Subtract each video frame in the video frame sequence from a previous video frame of each video frame to obtain a frame difference image; The frame difference image is binarized using a preset threshold to obtain a binarized image, and the moving target is determined from the binarized image.
7. The method according to claim 1, characterized in that The method for determining the annotation area includes: determining the annotation area based on a target detection model; determining the target annotation area where the object corresponding to the target annotation category is located from the image to be annotated based on the method for determining the annotation area includes: Using a target detection model for detecting objects corresponding to the target annotation category, target detection is performed on the image to be annotated to obtain a detection object; wherein the target detection model is trained by annotating image data of objects of the target annotation category; The image region where the detected object is located is determined as the target annotated region where the object corresponding to the target annotated category is located, thereby realizing the determination of the annotated region based on the target detection model.
8. The method according to claim 7, characterized in that After constructing the annotation data according to the target annotation area of the image to be annotated and the target annotation category, the method further includes: Using the annotation data, expanding the image data of the object annotated with the target annotation category to obtain expanded image data; The object detection model is retrained using the expanded image data to obtain an optimized object detection model.
9. The method according to any one of claims 1 to 8, characterized in that: After determining the target annotation region where the object corresponding to the target annotation category is located from the image to be annotated based on the annotation region determination method, and before constructing the annotation data based on the target annotation region of the image to be annotated and the target annotation category, the method further includes: In the image to be annotated, annotating the target annotation area, and displaying the annotated image to be annotated; When an adjustment operation on the target marked area is detected, in response to the adjustment operation, the target marked area is adjusted to obtain an adjusted target marked area; The constructing of the annotation data according to the target annotation area of the image to be annotated and the target annotation category includes: The labeling data is constructed using the adjusted target labeling area and the target labeling category.
10. The method according to any one of claims 1 to 8, characterized in that: The constructing of the annotation data according to the target annotation area of the image to be annotated and the target annotation category includes: From the image to be annotated, intercepting the image portion corresponding to the target annotation area to obtain a intercepted image; The target annotation category and the captured image are integrated into the annotation data.
11. A data labeling device, characterized in that: include: A data acquisition module, used to acquire the image to be annotated and the target annotation category; wherein the target annotation category represents the category of the area where the corresponding object is to be annotated; A labeling method determination module, used for, when the image to be labeled contains an object of a geometric shape, determining the labeling area in a method of determining the labeling area based on template matching; when the image to be labeled is a video frame sequence, determining the labeling area in a method of determining the labeling area based on target tracking, or determining the labeling area based on moving target detection; when the image to be labeled contains an object of a shape other than the geometric shape, determining the labeling area in a method of determining the labeling area based on a target detection model; A data annotation module, configured to determine, based on the annotation area determination method, a target annotation area where an object corresponding to the target annotation category is located from the image to be annotated; The data construction module is used to construct the annotation data according to the target annotation area of the image to be annotated and the target annotation category.
12. A data annotation device, characterized in that: include: A memory, for storing executable data annotation instructions; A processor, configured to implement the method according to any one of claims 1 to 10 when executing the executable data annotation instructions stored in the memory.
13. A computer-readable storage medium, characterized in that: Executable data annotation instructions are stored, and when executed by a processor, the method described in any one of claims 1 to 10 is implemented.
