Label data acquisition method and device, computer device and storage medium

Abstract

The application relates to a data labeling acquisition method and device, computer equipment, a storage medium and a computer program product. The method is applied to a labeling request end and comprises the following steps: acquiring attribute information of a target model class; constructing an instance of the target model class according to the attribute information; sending a data labeling request corresponding to to-be-labeled data to a data labeling end based on a labeling request method in the instance, so that the data labeling end labels the to-be-labeled data as labeled data according to the data labeling request; sending a result query request indicating whether the labeled data exists to the data labeling end based on a result download method in the instance; if a labeled data existence response returned by the data labeling end in response to the result query request is received, a result download request is sent to the data labeling end, and the labeled data returned by the data labeling end in response to the result download request is received. The method can improve the acquisition efficiency of the labeling request end for the labeled data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data interaction, and in particular to a method and apparatus for obtaining labeled data, a computer device, a storage medium, and a computer program product. BACKGROUND

[0002] In the field of artificial intelligence, various models are used to implement natural language processing, image recognition, intelligent recommendation, and the like. Since the models need to be optimized and adjusted using labeled data sets (such as training sets and test sets) constructed from labeled data, in order to improve the performance of the models, the labeled data needs to be prepared in advance. In the process of data labeling, interaction between a labeling request end and a data labeling end is involved. The labeling request end is used to send a labeling request and obtain labeled data, and the data labeling end is used to label data.

[0003] In the labeling request end, the traditional method for obtaining labeled data requires manual sending of data to be labeled to the data labeling end, manual checking of whether the data labeling end has completed the data labeling work, and manual downloading of the labeled data completed by the data labeling end.

[0004] As can be seen, the traditional technology requires a lot of human cost, resulting in low efficiency of the labeling request end in obtaining labeled data. SUMMARY

[0005] Therefore, it is necessary to provide a method and apparatus for obtaining labeled data, a computer device, a computer readable storage medium, and a computer program product, which can improve the efficiency of the labeling request end in obtaining labeled data.

[0006] In a first aspect, the present application provides a method for obtaining labeled data. The method is applied to a labeling request end and includes the following steps.

[0007] obtaining attribute information of a target model class;

[0008] constructing an instance of the target model class according to the attribute information;

[0009] sending, based on a labeling request method in the instance, a data labeling request corresponding to data to be labeled to a data labeling end, so that the data labeling end labels the data to be labeled as labeled data according to the data labeling request;

[0010] sending, based on a result downloading method in the instance, a result query request indicating whether the labeled data exists to the data labeling end, and if a labeled data existence response returned by the data labeling end in response to the result query request is received, sending a result downloading request to the data labeling end and receiving the labeled data returned by the data labeling end in response to the result downloading request.

[0011] In one of the embodiments, the method further comprises:

[0012] based on the operation selection method in the instance, sending operation option information for the labeling data to a labeling request account of the labeling request end, and obtaining an operation instruction generated by the labeling request account in response to the operation option information;

[0013] If the operation instruction is a labeling continuation instruction, based on the labeling request method corresponding to the labeling continuation instruction in the instance, the data labeling end is continuously sent the data labeling request corresponding to the labeling continuation instruction.

[0014] In one of the embodiments, the method further comprises:

[0015] If the operation instruction is a labeling end instruction, based on the result verification method in the instance, it is verified whether the labeling data is qualified, and if the labeling data is qualified, a labeling data set is constructed according to the labeling data.

[0016] In one of the embodiments, the method further comprises:

[0017] The labeling request message corresponding to the data labeling request is sent to the data labeling account of the data labeling end;

[0018] and / or,

[0019] The operation selection message corresponding to the operation option information is sent to the labeling request account of the labeling request end.

[0020] In one of the embodiments, the method further comprises:

[0021] Based on the data splitting method in the instance, the initial to-be-labeled data is split according to the labeling type to obtain to-be-labeled data;

[0022] The data labeling request corresponding to the to-be-labeled data is sent to the data labeling end based on the labeling request method in the instance, so that the data labeling end labels the to-be-labeled data as labeling data according to the data labeling request, comprising:

[0023] Based on the labeling request method in the instance, the to-be-labeled data and the labeling requirement information are used to generate a labeling task, and a data labeling request carrying the labeling task is sent to the data labeling end, so that the data labeling end labels the to-be-labeled data as labeling data according to the labeling task in the data labeling request.

[0024] In one of the embodiments, the attribute information of the target model class is obtained, comprising:

[0025] Obtain the specific attribute values ​​of the target model class input by the annotation request account through the annotation request terminal;

[0026] Read the pre-stored base model class's base properties, base property values, and specific properties;

[0027] The specific attribute value is assigned to a specific attribute of the base model class, so that the base model class is transformed into the target model class;

[0028] The basic attributes, basic attribute values, specific attributes, and specific attribute values ​​in the target model class are determined as the attribute information of the target model class.

[0029] Secondly, this application also provides a device for acquiring annotation data. The device is used for an annotation request end and includes:

[0030] The information acquisition module is used to acquire attribute information of the target model class;

[0031] An instance construction module is used to construct an instance of the target model class based on the attribute information;

[0032] The annotation request module is used to send a data annotation request corresponding to the data to be annotated to the data annotation end based on the annotation request method in the example, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request;

[0033] The result download module is used to send a result query request indicating whether the labeled data exists to the data annotation terminal based on the result download method in the example. If the data annotation terminal responds to the result query request with a response indicating that the labeled data exists, the module sends a result download request to the data annotation terminal and receives the labeled data returned by the data annotation terminal in response to the result download request.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0035] Obtain the attribute information of the target model class;

[0036] Based on the attribute information, construct an instance of the target model class;

[0037] Based on the annotation request method in the example, the data annotation request corresponding to the data to be annotated is sent to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request;

[0038] Based on the result download method in the example, a result query request indicating whether the labeled data exists is sent to the data annotation terminal. If a response indicating the existence of labeled data is received from the data annotation terminal in response to the result query request, a result download request is sent to the data annotation terminal, and the labeled data returned by the data annotation terminal in response to the result download request is received.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Obtain the attribute information of the target model class;

[0041] Based on the attribute information, construct an instance of the target model class;

[0042] Based on the annotation request method in the example, the data annotation request corresponding to the data to be annotated is sent to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request;

[0043] Based on the result download method in the example, a result query request indicating whether the labeled data exists is sent to the data annotation terminal. If a response indicating the existence of labeled data is received from the data annotation terminal in response to the result query request, a result download request is sent to the data annotation terminal, and the labeled data returned by the data annotation terminal in response to the result download request is received.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Obtain the attribute information of the target model class;

[0046] Based on the attribute information, construct an instance of the target model class;

[0047] Based on the annotation request method in the example, the data annotation request corresponding to the data to be annotated is sent to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request;

[0048] Based on the result download method in the example, a result query request indicating whether the labeled data exists is sent to the data annotation terminal. If a response indicating the existence of labeled data is received from the data annotation terminal in response to the result query request, a result download request is sent to the data annotation terminal, and the labeled data returned by the data annotation terminal in response to the result download request is received.

[0049] The aforementioned method, apparatus, computer equipment, storage medium, and computer program product for acquiring labeled data construct instances based on the attribute information of the target model class. Based on the annotation request method within the instance, a data annotation request is automatically generated and sent to the data annotation end. Furthermore, based on the result download method within the instance, a result query request is automatically sent to the data annotation end. Upon receiving a response indicating the existence of labeled data, a result download request is automatically triggered, thereby obtaining the labeled data from the data annotation end. It can be understood that this application, by constructing instances of the target model class and calling the methods within those instances, can automatically execute the method for acquiring labeled data on the annotation request end in a pipeline manner, reducing user operations and improving the efficiency of the annotation request end in acquiring labeled data. Attached Figure Description

[0050] Figure 1 This is an application environment diagram of a method for obtaining labeled data in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a method for obtaining labeled data in one embodiment;

[0052] Figure 3 This is a structural block diagram of a device for acquiring labeled data in one embodiment;

[0053] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The method for obtaining annotation data provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the annotation request end 102 communicates with the data annotation end 104 via a network. Specifically, the annotation request end 102 obtains the attribute information of the target model class and constructs an instance of the target model class based on this attribute information. The annotation request end 102 sends the data annotation request corresponding to the data to be annotated to the data annotation end 104 based on the annotation request method in the instance. The data annotation end 104 annotates the data to be annotated as labeled data according to the data annotation request. The annotation request end 102 sends a result query request indicating whether the labeled data exists to the data annotation end 104 based on the result download method in the instance. If the data annotation end 104 responds to the result query request by returning a response indicating that the labeled data exists to the annotation request end 102, then the annotation request end 102 sends a result download request to the data annotation end 104. The data annotation end 104 responds to the result download request by returning the labeled data to the annotation request end 102.

[0056] The annotation request terminal 102 and the data annotation terminal 104 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The annotation request terminal 102 and the data annotation terminal 104 can also be implemented using independent servers or server clusters composed of multiple servers.

[0057] In one embodiment, such as Figure 2 As shown, a method for obtaining labeled data is provided, and this method is applied to... Figure 1 Taking the annotation request terminal 102 as an example, the explanation includes the following steps:

[0058] Step S202: Obtain the attribute information of the target model class.

[0059] Here, a model class refers to a class specific to a model. In this embodiment, a class describes a collection of objects in the model that share the same attributes and methods. The class defines the attributes and methods common to each object in the collection. In object-oriented programming software (such as Python), model classes can be created using the `class` statement.

[0060] Specifically, the annotation requester obtains the attribute information of the target model class.

[0061] Step S204: Construct an instance of the target model class based on the attribute information.

[0062] Specifically, the annotation request client instantiates the target model class based on the attribute information, obtaining an instance of the target model class. This instance is configured with abstract methods such as annotation request methods, result download methods, and operation selection methods for acquiring annotation data.

[0063] Step S206: Based on the annotation request method in the example, send the data annotation request corresponding to the data to be annotated to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request.

[0064] The data to be labeled can be images, text, etc.

[0065] Specifically, the annotation request client calls the annotation request method in the instance, and based on this annotation request method, sends the data annotation request corresponding to the data to be annotated to the data annotation client. Optionally, the annotation request client generates a data annotation request based on the data to be annotated, thus the data annotation request carries the data to be annotated. Optionally, the annotation request client generates a data annotation request based on the storage directory corresponding to the data to be annotated, thus the data annotation request carries the storage directory, and the data annotation client can then load the data to be annotated based on that storage directory. The data annotation client receives and parses the data annotation request, thereby annotating the data to be annotated as labeled data.

[0066] Step S208: Based on the result download method in the example, send a result query request to the data annotation terminal indicating whether the annotation data exists. If a response is received from the data annotation terminal indicating that the annotation data exists in response to the result query request, send a result download request to the data annotation terminal and receive the annotation data returned by the data annotation terminal in response to the result download request.

[0067] In this context, labeled data can be viewed as the result obtained after labeling the data to be labeled. A result query request is used to check if the labeled data exists. A response indicating that the labeled data exists indicates that the labeled data does exist.

[0068] Specifically, after the annotation requester sends a data annotation request, it calls the result download method in the instance. Based on this method, it sends a result query request indicating the existence of the annotation data to the data annotation client at preset time intervals (e.g., 10 minutes). If the data annotation client responds to the result query request with a "not found" response, it indicates that the annotation is not complete, and it waits for the preset time interval before continuing to send result query requests indicating the existence of the annotation data to the data annotation client. If the data annotation client responds to the result query request with a "does exist" response, it indicates that the annotation is complete, and the annotation requester sends a result download request to the data annotation client. The data annotation client, based on this result download request, returns the completed annotation data to the annotation requester. The annotation requester then downloads the annotation data to a preset path.

[0069] In the above method for obtaining labeled data, an instance is constructed based on the attribute information of the target model class. A data labeling request is automatically generated and sent to the data labeling end based on the labeling request method within the instance. Furthermore, a result query request is automatically sent to the data labeling end based on the result download method within the instance. Upon receiving a response indicating the existence of labeled data, a result download request is automatically triggered, thereby obtaining the labeled data from the data labeling end. It can be understood that this method, by constructing an instance of the target model class and calling the methods within that instance, can automatically execute the method for obtaining labeled data on the labeling request end in a pipeline manner, reducing user operations and improving the efficiency of the labeling request end in obtaining labeled data.

[0070] In one embodiment, a selection process is involved regarding whether the labeled data returned by the data annotation terminal needs to continue annotation. Based on the above embodiment, the method further includes the following steps:

[0071] Step S212: Based on the operation selection method in the example, send operation option information for the annotation data to the annotation request account of the annotation request end, and obtain the operation instructions generated by the annotation request account in response to the operation option information;

[0072] Step S214: If the operation instruction is a labeling continuation instruction, then based on the labeling request method corresponding to the labeling continuation instruction in the instance, continue to send the data labeling request corresponding to the labeling continuation instruction to the data labeling end.

[0073] Specifically, after receiving the annotation data returned by the data annotation end, the annotation request end calls the operation selection method in the instance and, based on this operation selection method, sends operation option information for the annotation data to the annotation request account of the annotation request end. After the annotation request account selects the target operation option, the annotation request end obtains the operation instruction corresponding to the target operation option. If the operation instruction is an annotation continuation instruction, the annotation request end, based on the annotation request method corresponding to the annotation continuation instruction in the instance, continues to send the data annotation request corresponding to the annotation continuation instruction to the data annotation end, so that the data annotation end continues to complete the data annotation work according to the data annotation request.

[0074] In a specific example, after obtaining the labeled data, the annotation requester sends operation options information for the labeled data to the annotation request account. These options include annotation continuation operations (e.g., re-annotate (incrementing the annotation round), reprocessing (incrementing both the processing and annotation rounds), final round operations (performing the final round of data annotation), and annotation end operations (stopping annotation (calling subsequent data processing methods in the instance)). The processing round can be understood as the number of data processing rounds, the purpose of which is to initially process the data to be labeled into a dataset suitable for model processing (e.g., validation). A processing round can contain multiple annotation rounds. The annotation round can be understood as the number of data annotation rounds. The annotation requester can generate a annotation continuation instruction based on the selected annotation continuation operation. The annotation requester can generate an annotation end instruction based on the selected annotation end operation.

[0075] In this embodiment, by sending operation option information to the annotation request account and continuing the operation based on the annotation selected by the annotation request account, the cyclical processing of data annotation is realized, which enables the data annotation process to be flexibly controlled.

[0076] In one embodiment, in conjunction with the previous embodiment, the method further includes the following steps:

[0077] Step S222: If the operation instruction is the annotation end instruction, then the annotation data is verified to be qualified based on the result verification method in the instance. If the annotation data is qualified, then the annotation dataset is constructed based on the annotation data.

[0078] Specifically, if the operation instruction is a labeling end instruction, meaning the labeling request account has selected the labeling end operation, then the labeling request client calls the result verification method in the instance to verify whether the labeling data is qualified. If the labeling data is qualified, then a labeling dataset is constructed based on the labeling data. If the labeling data is unqualified, then the process returns to step S206.

[0079] In this embodiment, the annotation process can be flexibly controlled by allowing the annotation to continue or end based on the annotation request account's selected annotation. Annotations can be repeated or interrupted.

[0080] In one embodiment, the method involves sending interactive messages to the data annotation account and the annotation request account to notify the corresponding user to process the request in a timely manner. Based on the above embodiment, the method further includes the following steps:

[0081] Step S232: Send the annotation request message corresponding to the data annotation request to the data annotation account on the data annotation end;

[0082] And / or,

[0083] Step S234: Send the operation selection message corresponding to the operation option information to the annotation request account on the annotation request end.

[0084] Specifically, the annotation requester can send the corresponding annotation request message to the data annotation account of the data annotation end when the data annotation request is generated. The annotation requester can also send the corresponding annotation request message to the data annotation account of the data annotation end when the data annotation request is issued. Furthermore, the annotation requester can send the corresponding annotation request message to the data annotation account of the data annotation end after the data annotation request is issued. This notifies the data annotation account to complete the data annotation work in a timely manner. For example, the annotation request message can be sent to the data annotation end's Lark group and @ the corresponding data annotation account, thereby achieving the purpose of notifying the annotation personnel. It is understood that this embodiment does not limit the timing of sending the annotation request message.

[0085] When the annotation requester sends operation option information for the annotation data to its own annotation request account, it sends an operation selection message corresponding to the operation option information to that annotation request account. After sending the operation option information for the annotation data to its own annotation request account, the annotation requester sends an operation selection message corresponding to the operation option information to that annotation request account. For example, the operation selection message can be sent to the annotation requester's Lark group and @ the corresponding annotation request account, thereby notifying the annotation requester to complete the operation selection in a timely manner. It is understood that this embodiment does not limit the timing of sending the operation selection message.

[0086] In this embodiment, timely interaction with relevant personnel is achieved through message notifications. For example, a communication connection is established with Airflow via the API (Application Programming Interface) provided by Lark Robot, allowing Airflow to promptly deliver notification messages to users, reminding them to handle relevant tasks as soon as possible, thus improving processing efficiency. Furthermore, Lark is a next-generation enterprise office software that not only facilitates information exchange among colleagues, but also, compared to other competing products, allows for convenient information exchange between the software system and individuals through Lark Robot.

[0087] In one embodiment, the method further includes the following steps:

[0088] Step S242: Based on the data splitting method in the example, the initial data to be labeled is split according to the labeling type to obtain the data to be labeled.

[0089] Taking the image to be labeled as an example, the labeling type can be a bounding box, such as a human body bounding box, a face bounding box, a vehicle bounding box, etc., or an image label, such as whether the image is blurry, whether there is a face, whether there is a vehicle, etc., or an image comparison, such as a main image with a face and multiple sub-images with faces, and labeling which sub-images and the main image are of the same person.

[0090] Specifically, the annotation request client first copies the initial data to be labeled to a preset server path. Then, the annotation request client calls the data splitting method in the example, and based on this data splitting method, splits the initial data to be labeled according to the labeling type to obtain the split data to be labeled. It can be understood that in this embodiment, there are multiple data sets to be labeled. For example, the annotation request client splits the initial image to be labeled according to the detection box type in the initial image to be labeled, combining images containing human bounding boxes, combining images containing face bounding boxes, and combining images containing vehicle bounding boxes to obtain three types of data to be labeled.

[0091] Optionally, after the data is split, the data to be labeled is organized into the data format required for labeling.

[0092] In this embodiment, since a large amount of data can easily affect the stability of the data annotation end, before sending the data to be annotated, the initial large amount of data to be annotated is split according to the annotation type to construct small amounts of data to be annotated for different annotation types, which is beneficial for subsequent annotation personnel to import and annotate.

[0093] Furthermore, in one embodiment, step S206 can be implemented through the following steps:

[0094] Step S2062: Based on the annotation request method in the example, an annotation task is generated according to the data to be annotated and the annotation requirement information. The data annotation request carrying the annotation task is sent to the data annotation end so that the data annotation end can annotate the data to be annotated as annotated data according to the annotation task in the data annotation request.

[0095] The annotation requirements information may include annotation type, annotation personnel information, etc.

[0096] Specifically, the annotation request client calls the annotation request method in the instance. Based on the preset annotation rules in this method, it uses a labeling tool to generate an annotation task according to the data to be annotated and the annotation requirements. Based on this annotation task, it generates a data annotation request and sends the data annotation request carrying the annotation task to the data annotation client. Furthermore, after the annotation task is successfully created, the annotation request message corresponding to the data annotation request carrying the annotation task is sent to the data annotation account on the data annotation client. For example, the annotation request message is sent to a Lark group and @ the corresponding data annotation account (this data annotation account is pre-associated with the annotation type), thereby notifying the annotators. The data annotation client receives the data annotation request, parses it to obtain the annotation task, and then annotates the data to be annotated as labeled data according to the annotation task.

[0097] In one embodiment, step S202 can be implemented through the following steps:

[0098] Step S2022: Obtain the specific attribute values ​​of the target model class input by the annotation request account through the annotation request terminal;

[0099] Step S2024: Read the pre-stored basic attributes, basic attribute values, and specific attributes of the basic model class;

[0100] Step S2026: Assign a specific attribute value to a specific attribute of the base model class so that the base model class is transformed into the target model class;

[0101] Step S2028: Determine the basic attributes, basic attribute values, specific attributes, and specific attribute values ​​in the target model class as the attribute information of the target model class.

[0102] The base model class inherits its basic attributes and corresponding values ​​from the target model class. The base model class also includes abstract annotation request methods, result download methods, etc. Specific attributes and their values ​​are unique to the target model class.

[0103] Specifically, the annotation request client pre-stores a base model class. This base model class is configured with base attributes, corresponding base attribute values, and specific attributes. The annotation request account can input specific attribute values ​​of the target model class through the annotation request client's display interface, and then assign these specific attribute values ​​to specific attributes of the base model class, thus transforming the base model class into the target model class. The annotation request client then determines the base attributes, base attribute values, specific attributes, and specific attribute values ​​from the target model class as the attribute information of the target model class.

[0104] Optionally, the method for obtaining the labeled data can run within Airflow. Airflow is a development platform that enables the creation, scheduling, and monitoring of tasks programmatically. By triggering the external parameter passing mechanism of the Directed Acyclic Graph (DAG) through Airflow, specific attribute values ​​of the target model class can be obtained from the annotation request account's input via the Airflow front-end page on the annotation request side. Optionally, these specific attributes include the target model class type, processing round, annotation round, etc. Different data splitting methods and annotation request methods can be determined based on the processing round and annotation round.

[0105] In this embodiment, by pre-setting a base model class, the attribute information of the base model class can be quickly inherited when constructing the target model class, and specific attribute values ​​can be assigned to the base model class to obtain the target model class, thereby quickly determining the attribute information of the target model class.

[0106] The following describes an embodiment of this application using a specific application scenario.

[0107] Take the pedestrian re-identification (reid) model as an example.

[0108] The data splitting method involves downloading the initial unlabeled images from the online feed, which may include videos and images in .pb (Protocol Buffers) format. These initial unlabeled images are then parsed and classified according to specific file types, enabling them to be split and classified based on a unique pid (PersonID). Since this classification result is an output of the business line's algorithm engineering logic, it may contain errors, thus requiring further cleanup processes.

[0109] The first round of cleaning: To ensure that each PID folder contains only one image for a given person, the images under each PID need to be cleaned. The cleaning method involves first identifying the highest quality image in the PID folder as the main image, representing that PID. Then, checking if the other images under each PID depict the same person as the main image. The processed images are then formatted to meet the annotation requirements and manually annotated. Finally, based on the manually annotated images, any images under each PID that differ from the main image are cleaned up.

[0110] The second round of merging: Since the classification based on algorithmic engineering logic may assign the same person to different process IDs (PIDs), all PIDs need to be merged according to preset merging conditions. Specifically, the REID algorithm model is used to extract features from all images to be labeled. Then, the average features of each PID are calculated based on the PID classification. Finally, the feature distance between each PID and other PIDs is calculated (feature distance characterizes the dissimilarity between two feature matrices; the larger the distance, the less similar the two people are). Based on the feature distance, information about PIDs that are similar to each PID can be obtained. The main images of these PIDs are clustered together to form the final image to be labeled. Finally, the PIDs that can be merged are merged together with the labeled images based on the manually labeled images.

[0111] The third round of cleaning: Similar to the first round of cleaning, it is mainly to eliminate the errors caused by the second round of merging results, and can also confirm the cleanliness of the data a second time.

[0112] Fourth round of data validation: After the first three rounds of data processing, the labeled image can be considered to meet the requirements. These labeled images are then used to infer and evaluate the ReID model, and the performance of the ReID model on this dataset and the status of some erroneous use cases are checked. If the data meets the expected requirements, it is then organized and uploaded to the dataset platform; if there are still deviations, the processing is restarted from the first round.

[0113] To address the issues of long dataset construction cycles and high manpower costs, this application presents a streamlined dataset construction pipeline system. Based on the scheduling methods provided by the open platform Airflow, the processes for constructing datasets for various algorithm models are summarized and optimized. Different workflows are designed according to processing requirements, enabling workflow scheduling. The system automates the manual annotation process, including annotation submission and result processing, forming a complete automated dataset construction pipeline. This makes dataset construction work for algorithm engineers and testers controllable, simpler, and more efficient.

[0114] In this embodiment, the construction of the labeled dataset is fully automated, greatly improving work efficiency. It overcomes the constraint of Airflow's non-looping nature, allowing for flexible process control that is repeatable and interruptible. Each dataset's construction can be initiated using a pipeline, each completely independent, much like running separate processes. Therefore, multiple datasets can run pipelines simultaneously, achieving concurrency. Through multi-round processing and labeling, if the labeling is inaccurate after one round, it can continue to the second and third rounds until the labeling results meet the accuracy requirements, at which point it can stop. Therefore, multiple tasks for constructing data of different algorithm types can be executed concurrently, and the system's data labeling fault tolerance is strong. Using Airflow as the foundational tool for the pipeline, and given Airflow's support for distributed deployment, worker nodes can be deployed in a distributed manner, maximizing load balancing. The system has strong compatibility and an open interface design, meeting the dataset construction needs of different algorithm models. Unified management of related data ensures the accuracy and security of data processing.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] Based on the same inventive concept, this application also provides a device for acquiring annotation data to implement the above-described method for acquiring annotation data. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the annotation data acquisition device provided below can be found in the limitations of the annotation data acquisition method described above, and will not be repeated here.

[0117] In one embodiment, such as Figure 3 As shown, a device for acquiring labeled data is provided, including: an information acquisition module 302, an instance construction module 304, a labeling request module 306, and a result download module 308, wherein:

[0118] The information acquisition module 302 is used to acquire the attribute information of the target model class;

[0119] Instance building module 304 is used to construct an instance of the target model class based on attribute information;

[0120] The annotation request module 306 is used to send the data annotation request corresponding to the data to be annotated to the data annotation end based on the annotation request method in the instance, so that the data annotation end can annotate the data to be annotated as annotated data according to the data annotation request;

[0121] The result download module 308 is used to send a result query request indicating whether the labeled data exists to the data annotation end based on the result download method in the instance. If it receives a response from the data annotation end in response to the result query request indicating that the labeled data exists, it sends a result download request to the data annotation end and receives the labeled data returned by the data annotation end in response to the result download request.

[0122] In the aforementioned device for acquiring labeled data, an instance is constructed based on the attribute information of the target model class. A data labeling request is automatically generated and sent to the data labeling end based on the labeling request method within the instance. Furthermore, a result query request is automatically sent to the data labeling end based on the result download method within the instance. Upon receiving a response indicating the existence of labeled data, a result download request is automatically triggered, thereby obtaining the labeled data from the data labeling end. It can be understood that this device, by constructing an instance of the target model class and calling the methods within that instance, can automatically execute the method for acquiring labeled data from the labeling request end in a pipeline manner, reducing user operations and improving the efficiency of the labeling request end in acquiring labeled data.

[0123] In one embodiment, the device further includes:

[0124] The operation selection module is used to send operation option information for the annotation data to the annotation request account on the annotation request side based on the operation selection method in the instance, and to obtain the operation instructions generated by the annotation request account in response to the operation option information.

[0125] The annotation continuation module is used to send the data annotation request corresponding to the annotation continuation instruction to the data annotation end based on the annotation request method corresponding to the annotation continuation instruction in the instance if the operation instruction is an annotation continuation instruction.

[0126] In one embodiment, the device further includes:

[0127] The annotation end module is used to verify whether the annotation data is qualified based on the result verification method in the instance if the operation command is the annotation end command; if the annotation data is qualified, the annotation dataset is constructed based on the annotation data.

[0128] In one embodiment, the device further includes:

[0129] The data splitting module is used to split the initial data to be labeled according to the labeling type based on the data splitting method in the instance, and obtain the data to be labeled.

[0130] The annotation request module 306 is specifically used to generate annotation tasks based on the annotation request method in the instance, according to the data to be annotated and the annotation requirement information, and send the data annotation request carrying the annotation task to the data annotation end, so that the data annotation end can annotate the data to be annotated as annotated data according to the annotation task in the data annotation request.

[0131] In one embodiment, the information acquisition module 302 is specifically used to acquire the specific attribute value of the target model class input by the annotation request account through the annotation request terminal; read the pre-stored basic attributes, basic attribute values ​​and specific attributes of the basic model class; assign the specific attribute value to the specific attribute of the basic model class so that the basic model class is transformed into the target model class; and determine the basic attributes, basic attribute values, specific attributes and specific attribute values ​​in the target model class as the attribute information of the target model class.

[0132] Each module in the aforementioned data acquisition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0133] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for acquiring labeled data. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0134] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0136] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0137] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for obtaining labeled data, characterized in that, The method is applied to the annotation request end and includes: Obtain the attribute information of the target model class and the corresponding data to be labeled; the target model class includes a collection of objects in the target model that have the same attribute information and methods; Based on the attribute information, an instance of the target model class is constructed; the instance includes an annotation request method and a result download method for implementing labeled data acquisition; the attribute information is determined by the basic attributes, basic attribute values, specific attributes, and specific attribute values ​​in the target model class, the specific attributes including processing rounds and annotation rounds, the processing rounds and the annotation rounds are used to determine different data splitting methods and annotation request methods in the instance; The initial data to be labeled is copied to a preset server path. Based on the data splitting method in the example, the initial data to be labeled is split according to the labeling type. The initial data to be labeled is then organized into the data format required for labeling to obtain the data to be labeled. Based on the annotation request method in the example, the data annotation request corresponding to the data to be annotated is sent to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request; Based on the result download method in the example, a result query request indicating whether the labeled data exists is sent to the data annotation terminal. If a response indicating the existence of labeled data is received from the data annotation terminal in response to the result query request, a result download request is sent to the data annotation terminal, and the labeled data returned by the data annotation terminal in response to the result download request is received. Based on the operation selection method in the example, operation option information for the annotation data is sent to the annotation request account of the annotation request end, and operation instructions generated by the annotation request account in response to the operation option information are obtained; If the operation instruction is a labeling continuation instruction, then based on the labeling request method corresponding to the labeling continuation instruction in the example, the data labeling request corresponding to the labeling continuation instruction is sent to the data labeling end.

2. The method according to claim 1, characterized in that, The method further includes: If the operation instruction is an annotation end instruction, then the annotation data is verified as qualified based on the result verification method in the example. If the annotation data is qualified, then an annotation dataset is constructed based on the annotation data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Send the annotation request message corresponding to the data annotation request to the data annotation account of the data annotation terminal; And / or, Send the operation selection message corresponding to the operation option information to the annotation request account of the annotation request end.

4. The method according to claim 1, characterized in that, The annotation request method based on the example sends the data annotation request corresponding to the data to be annotated to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request, including: Based on the annotation request method in the example, an annotation task is generated according to the data to be annotated and the annotation requirement information. A data annotation request carrying the annotation task is sent to the data annotation end, so that the data annotation end annotates the data to be annotated as annotated data according to the annotation task in the data annotation request.

5. The method according to claim 1, characterized in that, The acquisition of attribute information of the target model class includes: Obtain the specific attribute values ​​of the target model class input by the annotation request account through the annotation request terminal; Read the pre-stored base model class's base properties, base property values, and specific properties; The specific attribute value is assigned to a specific attribute of the base model class, so that the base model class is transformed into the target model class; The basic attributes, basic attribute values, specific attributes, and specific attribute values ​​in the target model class are determined as the attribute information of the target model class.

6. A device for acquiring labeled data, characterized in that, The device is used for labeling request terminals and includes: The information acquisition module is used to acquire the attribute information of the target model class and the data to be labeled corresponding to the target model class; the target model class includes a collection of objects in the target model that have the same attribute information and methods; An instance construction module is used to construct an instance of the target model class based on the attribute information; the instance includes an annotation request method and a result download method for implementing labeled data acquisition; the attribute information is determined by the basic attributes, basic attribute values, specific attributes, and specific attribute values ​​in the target model class, the specific attributes including processing rounds and annotation rounds, the processing rounds and the annotation rounds are used to determine different data splitting methods and annotation request methods in the instance; The data splitting module is used to copy the initial data to be labeled to a preset server path. Based on the data splitting method in the example, the initial data to be labeled is split according to the labeling type, and the initial data to be labeled is organized into the data format required for labeling to obtain the data to be labeled. The annotation request module is used to send a data annotation request corresponding to the data to be annotated to the data annotation end based on the annotation request method in the example, so that the data annotation end annotates the data to be annotated as annotated data according to the data annotation request; The result download module is used to send a result query request indicating whether the labeled data exists to the data annotation terminal based on the result download method in the example. If the data annotation terminal responds to the result query request and returns a response indicating that the labeled data exists, the module sends a result download request to the data annotation terminal and receives the labeled data returned by the data annotation terminal in response to the result download request. The operation selection module is used to send operation option information for the annotation data to the annotation request account of the annotation request end based on the operation selection method in the example, and to obtain the operation instructions generated by the annotation request account in response to the operation option information. The annotation continuation module is used to send the data annotation request corresponding to the annotation continuation instruction to the data annotation terminal based on the annotation request method corresponding to the annotation continuation instruction in the example if the operation instruction is an annotation continuation instruction.

7. The apparatus according to claim 6, characterized in that, The device further includes: The annotation end module is used to verify whether the annotation data is qualified based on the result verification method in the example if the operation instruction is an annotation end instruction. If the annotation data is qualified, an annotation dataset is constructed based on the annotation data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images