Data annotation methods, devices, storage media, and electronic devices

Abstract

This application discloses a data annotation method, apparatus, storage medium, and electronic device. The method includes: determining the target data type corresponding to the dataset to be annotated; if no pre-trained recognition model corresponding to the target data type exists in the recognition model set, dividing the dataset to be annotated into a first dataset and a second dataset; responding to a first annotation instruction for the target object, annotating the first dataset, and training a recognition model of a predetermined type using the annotated first dataset to obtain an initial recognition model; iterating the initial recognition model based on the second dataset to obtain a first target recognition model, and inputting the second dataset into the first target recognition model to generate a visualized annotation result. This application solves the technical problems of cumbersome data annotation steps, large workload, long time consumption, and high repetition in related technologies.

Description

Technical Field

[0001] This application relates to the field of data annotation, and more specifically, to a data annotation method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the development of AI, various information and data are growing exponentially. Labeling datasets is a very important step for deep learning. However, labeling data is a very tedious and time-consuming task. In related technologies, labeling is generally done manually or by using some labeling tools. Using labeling tools can greatly reduce the workload, but these existing labeling tools have high requirements for training modes, require large amounts of labeled training sample data, and still have a large workload and high repetition.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data annotation method, apparatus, storage medium, and electronic device to at least solve the technical problems in related technologies such as cumbersome data annotation steps, large annotation workload, long time consumption, and high repetition.

[0005] According to one aspect of the embodiments of this application, a data annotation method is provided, comprising: acquiring a dataset to be annotated and determining the target data type corresponding to the dataset to be annotated; searching for a pre-trained recognition model corresponding to the target data type from a set of recognition models; if no pre-trained recognition model corresponding to the target data type exists in the set of recognition models, dividing the dataset to be annotated into a first dataset and a second dataset; responding to a first annotation instruction of a target object, annotating the first dataset, and training a recognition model of a predetermined type using the annotated first dataset to obtain an initial recognition model; iterating the initial recognition model based on the second dataset to obtain a first target recognition model, and inputting the second dataset into the first target recognition model to generate a visualized annotation result.

[0006] Optionally, obtaining a first target recognition model by iterating the initial recognition model based on the second dataset includes: dividing the second dataset into multiple sub-data sets; determining whether all sub-data sets have been labeled; if not all sub-data sets have been labeled, generating a new training set, wherein the new training set includes: the existing training set and the labeled data; and training the initial model based on the new training set to obtain the first target recognition model.

[0007] Optionally, the second dataset is input into the first target recognition model to generate visualized annotation results, including: identifying all images to be annotated included in the second dataset; performing inference enhancement processing on all images to be annotated; and inputting the inference-enhanced images to be annotated into the first target recognition model to obtain annotation results.

[0008] Optionally, the inference enhancement process can be implemented as follows: determine the display attributes of each image in all images to be labeled; adjust the display attributes of each image to obtain multiple images with different display attributes corresponding to each image, wherein the display attributes include at least: saturation.

[0009] Optionally, the image to be labeled after inference enhancement is input into the first target recognition model to obtain the labeling result, including: determining the confidence level of each pre-selected box in the image to be labeled after inference enhancement; comparing the confidence level with a preset threshold, filtering each pre-selected box based on the comparison result to obtain the target pre-selected box, and generating the labeling result after visualizing the target pre-selected box.

[0010] Optionally, the method further includes: if a pre-trained recognition model exists for the data to be labeled, dividing the dataset to be labeled into a first dataset and a second dataset; dividing the second dataset into multiple sub-data sets; determining whether all sub-data sets have been labeled; if not all sub-data sets have been labeled, generating a new training set, wherein the new training set includes: the existing training set and the labeled data; and training the pre-trained recognition model based on the new training set to obtain a second target recognition model.

[0011] Optionally, the method further includes: displaying the annotation results on an editable display interface; receiving a second annotation instruction from the target object on the editable display interface, wherein the second annotation instruction is used to modify the annotation results; modifying the annotation results based on the second annotation instruction; and storing the modified annotation results in a predetermined location.

[0012] According to another aspect of the embodiments of this application, a data annotation apparatus is also provided, comprising: an acquisition module, configured to acquire a dataset to be annotated and determine the target data type corresponding to the dataset to be annotated; a partitioning module, configured to search for a pre-trained recognition model corresponding to the target data type from a set of recognition models, and, if no pre-trained recognition model corresponding to the target data type exists in the set of recognition models, partition the dataset to be annotated to obtain a first dataset and a second dataset; an annotation module, configured to respond to a first annotation instruction of a target object, annotate the first dataset, and train a recognition model of a predetermined type using the annotated first dataset to obtain an initial recognition model; and a generation module, configured to iterate the initial recognition model based on the second dataset to obtain a first target recognition model, and input the second dataset into the first target recognition model to generate a visualized annotation result.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute any data annotation method.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any data annotation method.

[0015] In this embodiment, a method based on a recognition model is used to annotate the data to be labeled. This involves obtaining the dataset to be labeled and determining the target data type. A pre-trained recognition model corresponding to the target data type is searched from the recognition model set. If no pre-trained recognition model for the target data type exists in the set, the dataset to be labeled is divided into a first dataset and a second dataset. Responding to a first labeling instruction from the target object, the first dataset is labeled, and the labeled first dataset is used to train a recognition model of a predetermined type to obtain an initial recognition model. The initial recognition model is iterated based on the second dataset to obtain a first target recognition model, and the second dataset is input into the first target recognition model to generate a visualized labeling result. This achieves the goal of first checking if a pre-trained recognition model is directly available; if no available pre-trained recognition model is found, a recognition model is quickly built based on the dataset for labeling. This simplifies the data labeling process, reduces the workload, and avoids the repetitive creation of recognition models, thus solving the technical problems of cumbersome data labeling steps, large workload, long time consumption, and high repetitiveness in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a flowchart illustrating an optional data annotation method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating a data annotation method based on YOLOv5 in one embodiment of this application;

[0019] Figure 3 This is a flowchart of a data annotation method with a pre-trained recognition model in one embodiment of this application;

[0020] Figure 4 This is a system architecture diagram of a user-interactive intelligent annotation platform based on YOLOv5 deep learning in one embodiment of this application;

[0021] Figure 5 This is a schematic diagram of an optional data annotation device according to an embodiment of this application;

[0022] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to an embodiment of this application, an embodiment of a data annotation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] Figure 1 This refers to the data annotation method according to the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0027] Step S102: Obtain the dataset to be labeled and determine the target data type corresponding to the dataset to be labeled;

[0028] Step S104: Search for the trained recognition model corresponding to the target data type in the recognition model set. If there is no trained recognition model corresponding to the target data type in the recognition model set, divide the dataset to be labeled into the first dataset and the second dataset.

[0029] Step S106: In response to the first annotation instruction of the target object, the first dataset is annotated, and the annotated first dataset is used to train the recognition model of the predetermined type to obtain the initial recognition model;

[0030] Step S108: Based on the second dataset, iterate the initial recognition model to obtain the first target recognition model, and input the second dataset into the first target recognition model to generate a visualized annotation result.

[0031] This data annotation method involves obtaining the dataset to be annotated and determining the target data type corresponding to it. It then searches for a pre-trained recognition model corresponding to the target data type from the recognition model set. If no pre-trained recognition model for the target data type exists in the recognition model set, the dataset to be annotated is divided into a first dataset and a second dataset. Responding to the first annotation instruction of the target object, the first dataset is annotated, and the annotated first dataset is used to train a recognition model of a predetermined type to obtain an initial recognition model. Based on the second dataset, the initial recognition model is iterated to obtain a first target recognition model, and the second dataset is input into the first target recognition model to generate a visualized annotation result. This method achieves the goal of first checking if a pre-trained recognition model is directly available; if no available pre-trained recognition model is found, a recognition model is quickly built based on the dataset for annotation. This simplifies the data annotation process, reduces the annotation workload, and avoids the repeated creation of recognition models. It thus solves the technical problems of cumbersome data annotation steps, large annotation workload, long time consumption, and high repetition in related technologies.

[0032] It should be noted that the aforementioned predefined model types include, but are not limited to, the YOLOv5 model.

[0033] In some embodiments of this application, the first target recognition model is obtained by iterating the initial recognition model based on the second dataset. This can be achieved by dividing the second dataset into multiple sub-data sets; determining whether all sub-data sets have been labeled; and generating a new training set if not all sub-data sets have been labeled. The new training set includes the existing training set and the labeled data. The initial model is then trained based on the new training set to obtain the first target recognition model.

[0034] For example, the second dataset contains images in groups A, B, C, and D. First, the images in group A are input into the initial model (V1 version) for recognition, resulting in labeled result 'a'. This labeled result 'a' can then be modified manually, and the modified result 'a' is used as the training set for the initial model (V1 version) to train the initial model (V2 version). Next, the images in group B are input into the initial model (V2 version) for recognition, resulting in labeled result 'b'. This labeled result 'b' can then be manually modified, and the initial model (V2 version) is trained based on 'b' to train the initial model (V3 version). Similarly, the images in groups C and D are processed using the same iterative training logic to ultimately obtain a more accurate first target recognition model.

[0035] Understandably, after obtaining the first target recognition model, the second dataset can be re-identified to obtain the annotation results. The annotation results are then compared with the manually annotated results during the training process to determine the accuracy of the first target recognition model. If the model's recognition accuracy is poor, iterative training can continue until the accuracy meets the preset requirements.

[0036] For example, after obtaining the first target recognition model, images in groups A, B, C, and D can be input into the first target recognition model to obtain the annotation result 1 corresponding to each group of images. Then, the annotation result is compared with the annotation result 2 manually annotated during iterative training to determine the accuracy of the first target recognition model.

[0037] As an optional implementation, the second dataset is input into the first target recognition model to generate visualized annotation results. This can be achieved through the following steps: all images to be annotated in the second dataset are identified; inference enhancement processing is performed on all images to be annotated; and the inference-enhanced images are input into the first target recognition model to obtain annotation results.

[0038] Specifically, the above-mentioned reasoning enhancement processing can be achieved as follows: determine the display attributes of each image in all images to be labeled; adjust the display attributes of each image to obtain multiple images with different display attributes corresponding to each image. It should be noted that the above-mentioned display attributes can also be size, saturation, brightness, and contrast, etc.

[0039] In some embodiments of this application, the annotation result is obtained by inputting the inference-enhanced image to be annotated into the first target recognition model. This can be achieved in the following way: Specifically, the confidence level of each pre-selected box in the inference-enhanced image to be annotated can be determined; the confidence level can be compared with a preset threshold; based on the comparison result, each pre-selected box can be filtered to obtain the target pre-selected box; and the target pre-selected box can be visualized to generate the annotation result. For example, if an image to be annotated has three pre-selected boxes with confidence levels of 0.5, 0.7, and 0.9, respectively, and the preset threshold is 0.8, then the pre-selected box corresponding to 0.9 can be considered the target pre-selected box.

[0040] Understandably, inference-based augmentation can improve the accuracy of image annotation. For example, if the image to be annotated is a safety helmet, which typically comes in colors like red, blue, and yellow, then for the same image, photos with higher color saturation and brightness are easier to identify. Therefore, the attributes of the image to be annotated, such as brightness, saturation, and hue, can be actively adjusted to more accurately identify the target object. Similarly, if the image to be annotated is too small, the target object may be difficult to spot during the recognition process. Therefore, the size of the image to be annotated can be adjusted to facilitate accurate identification.

[0041] It should be noted that the above process is generally implemented through the program built into the first target recognition model. After the target pre-selection box is obtained, the annotation result is also automatically generated by the first target recognition model.

[0042] Figure 2 This is a flowchart illustrating a data annotation method based on YOLOv5 in one embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:

[0043] S21: Import dataset;

[0044] S22: Label N data points and provide label groups;

[0045] S23: Train using the YOLOv5 model to obtain a new model;

[0046] S24: Randomly select N data points from the dataset (unlabeled) for prediction.

[0047] S25: Perform inference enhancement by adjusting the image's size, saturation, hue, or contrast, and then apply a threshold to the IOU of each preselected box to obtain a more accurate preselected box.

[0048] S26: Label the data;

[0049] S27: Submit the labeled data to a human reviewer for correction;

[0050] S28: Determine whether all image annotations in the dataset are complete. If not, generate a new training set (the previous training set + the completed dataset and data cleaning) and proceed to steps S23-S27. Otherwise, enhance the model training with all data in the training set.

[0051] S29: Save and manage the trained model.

[0052] As an alternative implementation, if a pre-trained recognition model already exists for the data to be labeled, the dataset to be labeled is divided into a first dataset and a second dataset. The second dataset is then divided into multiple sub-data sets. It is determined whether all sub-data sets have been labeled. If not, a new training set is generated, which includes the existing training set and the labeled data. The pre-trained recognition model is then trained based on the new training set to obtain the second target recognition model. It is understood that if a pre-trained recognition model already exists in the model set for the data to be labeled, this pre-trained model can be directly retrieved from the model set, avoiding repeated training of the recognition model, saving labeling time, and improving labeling efficiency.

[0053] Figure 3 This is a flowchart of a data annotation method for a pre-trained recognition model in one embodiment of this application, such as... Figure 3 As shown, the method includes:

[0054] S31: Import the newly added dataset;

[0055] S32: Select the model and label group;

[0056] S33: Randomly select X data points from the dataset for prediction and label the data;

[0057] S34: Submit the annotated data to a human reviewer for correction;

[0058] S35: Determine whether all image annotations in the dataset are complete. If not, generate a new training set (the previous training set + the completed dataset and data cleaning) and proceed to steps S23-S27. Otherwise, enhance the model training with all data in the training set.

[0059] S36: Save and manage the trained model.

[0060] In some optional embodiments of this application, the annotation results are also displayed on an editable display interface; a second annotation instruction from the target object is received on the editable display interface, wherein the second annotation instruction is used to modify the annotation results; the annotation results are modified based on the second annotation instruction, and the modified annotation results are stored in a predetermined location.

[0061] Understandably, after the recognition model generates the labeled results, the labeled results of the dataset can be reviewed and modified manually to improve the accuracy of the labeled results. The modified labeled results can be used as the training set for the next training of the recognition model, updating the recognition model again and making the parameters of the recognition model more precise and the recognition results more accurate.

[0062] It is readily apparent that the solutions disclosed in the relevant embodiments of this application can effectively reduce the workload of manual annotation, reduce the complexity of dataset and model management, eliminate the conversion of labeled data, greatly improve the efficiency of intelligent data annotation, reduce repetitive manual work, and can quickly and accurately annotate large amounts of data.

[0063] In some embodiments of this application, a user-interactive intelligent annotation platform based on YOLOv5 deep learning is also provided. The intelligent annotation platform includes: a data and model management module, a data annotation module, an intelligent annotation module, and a model training module.

[0064] The data and model management module is used to manage user datasets and models. Specifically, in this embodiment, when a user uploads a dataset, the dataset is divided into labeled datasets and unlabeled datasets based on whether the dataset has labeled data, and the dataset version is controlled. The data is stored in the database. When the user labels or performs intelligent labeling on the dataset, the trained model is saved and managed, stored in the database, and the dataset and model are available for download.

[0065] The data annotation module is used by users to annotate images. Specifically, in this embodiment, users annotate images, obtain the position of the bounding box in the image, and automatically convert it into training data and store it in the corresponding image data. Users can perform add, delete, modify, and query operations on the annotation information in the image.

[0066] The intelligent annotation module is used by users to intelligently annotate datasets. This module can use the two intelligent annotation methods mentioned above. Specifically, in this embodiment, users intelligently annotate the already labeled datasets. The administrator divides the datasets into unlabeled data and labeled data, trains on the labeled data, annotates some of the unlabeled data, and submits the annotated data to the user for review, which then classifies it as an unreviewed dataset.

[0067] The model training module is used by users to enhance the training of the model on a fully labeled dataset. Specifically, in this embodiment, custom enhancement training is performed on the fully labeled dataset.

[0068] It is noteworthy that the online annotation method of this application can effectively reduce data transformation. Through the YOLOv5 model and inference enhancement, it can effectively improve the speed and accuracy of data detection. Moreover, it has low repetitiveness, as online annotation no longer requires repetitive data transformation. Iterative intelligent annotation increases the partitioning of the dataset exponentially, eliminating the need for excessive intelligent and manual annotation. In addition, the dataset and model are simple to manage. The dataset and model are stored in a database, which is convenient for management and use. Furthermore, it has low requirements for professional knowledge, and no extensive professional knowledge is required to annotate the dataset.

[0069] Figure 4 This is a system architecture diagram of a user-interactive intelligent annotation platform based on YOLOv5 deep learning in one embodiment of this application, as shown below. Figure 4 As shown, the system mainly consists of six parts: runtime environment, database, AI layer, data layer, business layer, and presentation layer. The structure of each part is as follows: Figure 4 As shown.

[0070] Figure 5 This is a data annotation device provided according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0071] Module 50 is used to acquire the dataset to be labeled and determine the target data type corresponding to the dataset to be labeled.

[0072] The partitioning module 52 is used to find the trained recognition model corresponding to the target data type from the recognition model set. If there is no trained recognition model corresponding to the target data type in the recognition model set, the dataset to be labeled is partitioned to obtain the first dataset and the second dataset.

[0073] The annotation module 54 is used to respond to the first annotation instruction of the target object, annotate the first dataset, and use the annotated first dataset to train a recognition model of a predetermined type to obtain an initial recognition model;

[0074] The generation module 56 is used to iterate the initial recognition model based on the second dataset to obtain the first target recognition model, and input the second dataset into the first target recognition model to generate a visualized annotation result.

[0075] In this data annotation device, the acquisition module 50 is used to acquire the dataset to be annotated and determine the target data type corresponding to the dataset; the partitioning module 52 is used to search for a pre-trained recognition model corresponding to the target data type from the recognition model set. If no pre-trained recognition model corresponding to the target data type exists in the recognition model set, the dataset to be annotated is partitioned into a first dataset and a second dataset; the annotation module 54 is used to respond to the first annotation instruction of the target object, annotate the first dataset, and use the annotated first dataset to train a recognition model of a predetermined type to obtain an initial recognition model; the generation module 56 is used to iterate the initial recognition model based on the second dataset to obtain a first target recognition model, and input the second dataset into the first target recognition model to generate a visualized annotation result. This achieves the goal of first checking whether a pre-trained recognition model can be directly used, and if no pre-trained recognition model is available, then quickly building a recognition model based on the dataset and annotating the dataset. This simplifies the data annotation steps, reduces the annotation workload, and avoids the technical effect of repeatedly building recognition models. It also solves the technical problems of cumbersome data annotation steps, large annotation workload, long time consumption, and high repetition in related technologies.

[0076] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute any data annotation method.

[0077] Specifically, the aforementioned storage medium is used to store program instructions for the following functions, thereby implementing the following functions:

[0078] Obtain the dataset to be labeled and determine the target data type corresponding to the dataset; search for the pre-trained recognition model corresponding to the target data type from the recognition model set; if no pre-trained recognition model corresponding to the target data type exists in the recognition model set, divide the dataset to be labeled into a first dataset and a second dataset; respond to the first labeling instruction of the target object, label the first dataset, and use the labeled first dataset to train the recognition model of the predetermined type to obtain an initial recognition model; iterate the initial recognition model based on the second dataset to obtain a first target recognition model, and input the second dataset into the first target recognition model to generate a visualized labeling result.

[0079] Optionally, in this embodiment, the storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of the storage medium include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0080] In an exemplary embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described data annotation methods.

[0081] Optionally, when executed by a processor, the computer program may perform the following steps:

[0082] Obtain the dataset to be labeled and determine the target data type corresponding to the dataset; search for the pre-trained recognition model corresponding to the target data type from the recognition model set; if no pre-trained recognition model corresponding to the target data type exists in the recognition model set, divide the dataset to be labeled into a first dataset and a second dataset; respond to the first labeling instruction of the target object, label the first dataset, and use the labeled first dataset to train the recognition model of the predetermined type to obtain an initial recognition model; iterate the initial recognition model based on the second dataset to obtain a first target recognition model, and input the second dataset into the first target recognition model to generate a visualized labeling result.

[0083] An electronic device is provided according to an embodiment of this application, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described data annotation methods.

[0084] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0085] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0086] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0087] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data annotation methods. For example, in some embodiments, the data annotation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data annotation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data annotation method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0094] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0095] In the relevant embodiments of this application, a method based on a recognition model is adopted to annotate the data to be labeled. This involves obtaining the dataset to be labeled and determining the target data type corresponding to it; searching for a pre-trained recognition model corresponding to the target data type from the recognition model set; if no pre-trained recognition model for the target data type exists in the recognition model set, dividing the dataset to be labeled into a first dataset and a second dataset; responding to a first labeling instruction from the target object, labeling the first dataset, and using the labeled first dataset to train a recognition model of a predetermined type to obtain an initial recognition model; iterating the initial recognition model based on the second dataset to obtain a first target recognition model, and inputting the second dataset into the first target recognition model to generate a visualized labeling result. This achieves the goal of first checking if a pre-trained recognition model is directly available, and if no available pre-trained recognition model is found, then quickly building a recognition model based on the dataset for labeling. This simplifies the data labeling steps, reduces the labeling workload, and avoids the repeated building of recognition models, thus solving the technical problems of cumbersome data labeling steps, large labeling workload, long time consumption, and high repetition in related technologies.

[0096] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0097] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0102] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data annotation method, characterized in that, include: Obtain the dataset to be labeled and determine the target data type corresponding to the dataset to be labeled; The trained recognition model corresponding to the target data type is searched from the recognition model set. If no trained recognition model corresponding to the target data type is found in the recognition model set, the dataset to be labeled is divided into a first dataset and a second dataset. In response to the first annotation instruction of the target object, the first dataset is annotated, and the annotated first dataset is used to train a recognition model of a predetermined type to obtain an initial recognition model; The process involves iterating the initial recognition model based on the second dataset to obtain a first target recognition model, and inputting the second dataset into the first target recognition model to generate visualized annotation results. This includes: dividing the second dataset into multiple sub-data sets; determining whether all sub-data sets have been fully annotated; if not, generating a new training set, wherein the new training set includes: an existing training set and annotated data, the annotated data being data annotated by the initial recognition model and corrected through manual review; and training the initial recognition model based on the new training set to obtain the first target recognition model.

2. The method according to claim 1, characterized in that, The second dataset is input into the first object recognition model to generate visualized annotation results, including: Identify all unlabeled images included in the second dataset; All the images to be labeled are subjected to inference enhancement processing, and the images to be labeled after inference enhancement processing are input into the first target recognition model to obtain the labeling results.

3. The method according to claim 2, characterized in that, The reasoning enhancement processing can be implemented in the following ways: Determine the display attributes of each image among all the images to be labeled; The display attributes of each image are adjusted to obtain multiple images with different display attributes corresponding to each image, wherein the display attributes include at least: saturation.

4. The method according to claim 2, characterized in that, The annotation result is obtained by inputting the inference-enhanced image to the first target recognition model, including: Determine the confidence level of each preselected box in the image to be labeled after the inference enhancement process; The confidence level is compared with a preset threshold, and each preselected box is filtered based on the comparison result to obtain the target preselected box. The annotation result is generated after the target preselected box is visualized.

5. The method according to claim 1, characterized in that, The method further includes: If the pre-trained recognition model exists in the data to be labeled, the dataset to be labeled is divided into a first dataset and a second dataset; The second dataset is divided into multiple sub-data sets; Determine whether all of the multiple sets of sub-data have been labeled. If not all of the multiple sets of sub-data have been labeled, generate a new training set, wherein the new training set includes: the existing training set and the labeled data. The pre-trained recognition model is trained based on the new training set to obtain a second target recognition model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The annotation results are displayed in an editable interface; Receive a second annotation instruction from the target object on the editable display interface, wherein the second annotation instruction is used to modify the annotation result; The annotation result is modified based on the second annotation instruction, and the modified annotation result is stored in a predetermined location.

7. A data annotation device, characterized in that, include: The acquisition module is used to acquire the dataset to be labeled and determine the target data type corresponding to the dataset to be labeled. The partitioning module is used to search for a trained recognition model corresponding to the target data type from the recognition model set. If there is no trained recognition model corresponding to the target data type in the recognition model set, the dataset to be labeled is partitioned into a first dataset and a second dataset. The annotation module is used to respond to the first annotation instruction of the target object, annotate the first dataset, and use the annotated first dataset to train a recognition model of a predetermined type to obtain an initial recognition model; A generation module is used to iterate the initial recognition model based on the second dataset to obtain a first target recognition model, and input the second dataset into the first target recognition model to generate a visualized annotation result. This includes: dividing the second dataset into multiple sub-data sets; determining whether all sub-data sets have been fully annotated; and if not, generating a new training set, wherein the new training set includes: an existing training set and annotated data, the annotated data being data annotated by the initial recognition model and corrected by manual review; and training the initial recognition model based on the new training set to obtain the first target recognition model.

8. A non-volatile storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the data annotation method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data annotation method as described in any one of claims 1 to 6.

Images