Data labeling method, display method, device, and electronic device

Abstract

The application discloses a data labeling method, a display method, a device, an electronic device and a computer readable storage medium. The data labeling method comprises the following steps: obtaining first labeling data of a first labeling client for a data labeling task, and adding the first labeling data to a to-be-detected data set; if the to-be-detected data set does not satisfy a preset condition, sending a first labeling request to a second labeling client, wherein the first labeling request comprises a request for labeling and analyzing the data labeling task; obtaining second labeling data of the second labeling client for the data labeling task and analysis content of the second labeling data, so as to display at least part of the analysis content in the second labeling client in the future; adding the second labeling data to the to-be-detected data set until the to-be-detected data set satisfies the preset condition; and determining a labeling result of the data labeling task according to the to-be-detected data set. The application can reduce the cost of data labeling and improve the efficiency and accuracy of the labeling result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data labeling method and display method, device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] Data labeling refers to the process of adding labels to each sample in a dataset, which can be used to train machine learning models to understand the content of the data. Currently, in order to ensure the accuracy of the labeling result, the common way on the market is that multiple labelers independently label the same data, and then determine the final labeling result through voting or expert review. However, due to individual judgment differences, different levels of professional knowledge and other factors, the current data labeling task often requires a large number of labelers to obtain relatively consistent labeling results, and the accuracy of the labeling result is low, and the efficiency is extremely low. SUMMARY

[0003] The present application provides a data labeling method, display method, device, electronic equipment and computer readable storage medium, which can reduce the cost of data labeling and improve the efficiency and accuracy of labeling results. The specific scheme is as follows:

[0004] In a first aspect, the embodiments of the present application provide a data labeling method, which comprises:

[0005] Obtaining first labeling data of a plurality of first labeling clients respectively for a data labeling task, and adding the first labeling data to a to-be-detected data set;

[0006] Determining whether the to-be-detected data set meets a preset condition;

[0007] When the to-be-detected data set does not meet the preset condition, sending a first labeling request for the data labeling task to a second labeling client, the first labeling request comprising a request for labeling the data labeling task and analyzing the labeled result;

[0008] Obtaining second labeling data of the second labeling client for the data labeling task and analysis content of the second labeling data, so as to display at least part of the analysis content in the second labeling client to be put in the future;

[0009] Adding the second labeling data to the to-be-detected data set, returning to execute the step of determining whether the to-be-detected data set meets the preset condition until the to-be-detected data set meets the preset condition;

[0010] According to the data in the to-be-detected data set, determining the labeling result of the data labeling task.

[0011] In a second aspect, an embodiment of the present application provides a display method, which comprises:

[0012] In response to receiving a first labeling request for a data labeling task, displaying, by a second labeling client, a labeling area and an analysis input area of the data labeling task, the first labeling request comprising a request for labeling the data labeling task and analyzing the labeled result;

[0013] In response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task; and

[0014] In response to an input operation in the analysis input area, obtaining analysis content input for the second labeling data, to display at least part of the analysis content in a second labeling client to be launched later.

[0015] In a third aspect, an embodiment of the present application provides a data labeling apparatus, which comprises:

[0016] A first obtaining unit, configured to obtain first labeling data for a data labeling task respectively by a plurality of first labeling clients, and add the first labeling data to a to-be-detected data set;

[0017] A first determining unit, configured to determine whether the to-be-detected data set meets a preset condition;

[0018] A sending unit, configured to send, when the to-be-detected data set does not meet the preset condition, a first labeling request for the data labeling task to a second labeling client, the first labeling request comprising a request for labeling the data labeling task and analyzing the labeled result;

[0019] A second obtaining unit, configured to obtain second labeling data for the data labeling task by the second labeling client, and analysis content for the second labeling data, to display at least part of the analysis content in a second labeling client to be launched later;

[0020] An adding unit, configured to add the second labeling data to the to-be-detected data set, and return to perform the step of determining whether the to-be-detected data set meets the preset condition, until the to-be-detected data set meets the preset condition;

[0021] A second determining unit, configured to determine a labeling result of the data labeling task according to data in the to-be-detected data set.

[0022] In a fourth aspect, an embodiment of the present application provides a display apparatus, which comprises:

[0023] The first display unit is configured to, in response to receiving a first labeling request for a data labeling task, display a labeling area and an analysis input area of the data labeling task through a second labeling client, the first labeling request including a request for labeling the data labeling task and analyzing the labeled result.

[0024] The third obtaining unit is configured to, in response to a labeling operation in the labeling area, obtain second labeling data for the data labeling task, and in response to an input operation in the analysis input area, obtain analysis content input for the second labeling data, and display at least part of the analysis content in a second labeling client to be launched later.

[0025] In a fifth aspect, the present application further provides an electronic device, comprising:

[0026] a processor; and

[0027] a memory configured to store a data processing program, and the electronic device, after being powered on and running the program through the processor, executes the method according to the first aspect or the second aspect.

[0028] In a sixth aspect, the present application further provides a computer readable storage medium storing a data processing program, and the program, when executed by a processor, executes the method according to the first aspect or the second aspect.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] The data labeling method provided by the present application comprises the following steps: obtaining first labeling data for a data labeling task from a plurality of first labeling clients; adding the first labeling data to a to-be-detected data set, and determining whether the to-be-detected data set meets a preset condition; when the to-be-detected data set does not meet the preset condition, sending a first labeling request for the data labeling task to a second labeling client, the first labeling request including a request for labeling the data labeling task and analyzing the labeled result; obtaining second labeling data for the data labeling task from the second labeling client and analysis content for the second labeling data, and displaying at least part of the analysis content in a second labeling client to be launched later; adding the second labeling data to the to-be-detected data set, returning to the step of determining whether the to-be-detected data set meets the preset condition until the to-be-detected data set meets the preset condition; and determining a labeling result of the data labeling task according to data in the to-be-detected data set.

[0031] As can be seen, the data annotation method provided in this application, when the first annotation data of the first annotation client for the data annotation task does not meet the preset conditions, requests the second annotation client to annotate while simultaneously providing an explanation of the annotation results when delivering the data annotation task to the second annotation client. Thus, upon obtaining the second annotation data and its parsed content from the second annotation client for the data annotation task, the parsed content is presented to subsequent second annotation clients. This assists subsequent second annotation clients in performing more accurate annotations using the parsed content as a reference. This allows for the rapid acquisition of increasingly consistent and highly accurate annotation data, thereby achieving rapid convergence of the data annotation task. Therefore, the data annotation method provided in this application can reduce the cost of data annotation and improve the efficiency and accuracy of annotation results. Attached Figure Description

[0032] Figure 1 This application provides a diagram of a labeling system for implementing a data labeling method;

[0033] Figure 2 This is a flowchart of the data annotation method provided in the first embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the entire process of the data annotation method provided in the first embodiment of this application;

[0035] Figure 4 This is a flowchart of the display method provided in the second embodiment of this application;

[0036] Figure 5 This is a schematic diagram of an example of the interface of the second annotation client in the display method provided in the second embodiment of this application;

[0037] Figure 6 This is a structural block diagram of an example of the data annotation device provided in the embodiments of this application;

[0038] Figure 7 This is a structural block diagram of an example of a display device provided in the embodiments of this application;

[0039] Figure 8 This is a structural block diagram of an example of an electronic device for data processing provided in an embodiment of this application. Detailed Implementation

[0040] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0041] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0042] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0043] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0044] Before describing the implementation methods of this application in detail, the prior art will be further explained first.

[0045] In the fields of machine learning and artificial intelligence, high-quality data annotation is fundamental to building accurate and reliable machine learning models. Data annotation refers to adding meaningful labels or annotations to raw data (such as images, audio, and text) to help algorithms understand key information within the data. With the rapid development of artificial intelligence, the importance of data annotation is becoming increasingly prominent.

[0046] Currently, traditional data annotation methods involve distributing data annotation tasks to a large number of dispersed annotators through data annotation platforms to improve the diversity and coverage of data annotation. Typically, to ensure the accuracy of the annotation results, multiple annotators independently annotate the same dataset, and then the final annotation result is determined by a majority vote mechanism or by expert review.

[0047] However, traditional data annotation methods face numerous challenges when dealing with highly subjective or specialized data annotation tasks. On one hand, even when a large number of independent annotation tasks are assigned to different annotators, differences in individual judgment and varying levels of expertise make it difficult to obtain highly confident and unique results. This significantly reduces the efficiency of data annotation and increases its cost. On the other hand, traditional data annotation methods primarily rely on simple majority voting or manual review, making it difficult to handle more complex or highly specialized data annotation tasks.

[0048] Therefore, how to reduce the cost of data annotation and improve the efficiency and accuracy of data annotation has become an urgent problem to be solved.

[0049] For the reasons mentioned above, in order to reduce the cost of data annotation and improve the efficiency and accuracy of data annotation, the first embodiment of this application provides a data annotation method. This method is applied to electronic devices, which may be desktop computers, laptops, mobile phones, tablets, electronic watches, etc., or other electronic devices capable of annotation. This embodiment of the application does not specifically limit the application.

[0050] For example, in conjunction with the above description, Figure 1 This application illustrates an annotation system 100 for implementing a data annotation method, which may include at least one annotation client 101, at least one server 102, and a network. Annotators can use the annotation client 101 to connect to different servers 102 via the network. The annotation client runs on any device with computing hardware capable of supporting and executing corresponding software application tools.

[0051] In the aforementioned annotation system 100, the annotation client 101 may be an application that supports annotation tasks. In some cases, the annotation client 101 may be a webpage, mini-program, or other client that can access the data annotation task. During the data annotation process, the annotation client 101 and the server 102 can interact.

[0052] In possible application scenarios, different annotation clients 101 may be served by different servers 102, and the servers 102 corresponding to different annotation clients 101 may be the same server.

[0053] In addition, when the annotation system 100 includes multiple annotation clients, multiple servers, and multiple networks, different annotation clients can connect to different servers through different networks.

[0054] The network can be a wireless or wired network, such as a wireless local area network (WLAN), local area network (LAN), cellular network, 2G network, 3G network, 4G network, 5G network, etc. Additionally, different annotation clients can connect to other annotation clients or to the server using their own Bluetooth network or hotspot network. Furthermore, the annotation system 100 can include multiple databases, which are coupled to different servers, and can store data related to data annotation in the databases.

[0055] It should be noted that, Figure 1 The illustrated annotation system diagram is merely an example. The annotation system 100 described in this application embodiment is intended to more clearly illustrate the technical solutions of this application embodiment and does not constitute a limitation on the technical solutions provided in this application embodiment.

[0056] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0057] The following, combined with Figure 2 and Figure 3 This application introduces a data annotation method provided in the first embodiment.

[0058] like Figure 2 As shown, the data annotation method provided in the first embodiment of this application includes the following steps S101 to S106.

[0059] Step S101: Obtain the first annotation data of multiple first annotation clients for the data annotation task, and add the first annotation data to the data set to be detected.

[0060] Data annotation is a crucial step in machine learning and artificial intelligence, particularly in training supervised learning models. The task of data annotation involves adding meaningful labels or tags to raw data so that machine learning models can learn patterns from these labels and make predictions or decisions. This data annotation task can include one of the following: image annotation, text annotation, speech annotation, video annotation, etc.

[0061] Specifically, when the data annotation task is an image annotation task, it can be a task of labeling the category of objects in an image, such as labeling the breed of a cat in an image; when the data annotation task is a text annotation task, it can be a task of labeling the sentiment category of text, such as labeling the sentiment category of the text "The weather is really nice today"; when the data annotation task is a speech annotation task, it can be a task of labeling the sentiment category of a speech or labeling entities in a speech, such as labeling the names of people, places, and actions in the speech "Xiaoming went to the museum today and visited the dinosaur fossils"; when the data annotation task is a video annotation task, it can be a task of recognizing the actions of objects in a video, such as labeling the number of shots each player made in a basketball game video.

[0062] In this embodiment of the application, the above-mentioned data annotation task can be first deployed to a preset number of first annotation clients. The preset number can be 2, 3, 4, etc., and this application does not limit this, so that the annotator holding the first annotation client can annotate the data annotation task. In this way, the first annotation data of each first annotation client for the data annotation task can be obtained, and the first annotation data is the annotation result of the first annotation client for the data annotation task.

[0063] Table 1 is an example table of the first annotation data of the first annotation client for the data annotation task in the data annotation method provided in the embodiments of this application.

[0064]

[0065] In Table 1 above, the data annotation task is an image annotation task, providing a picture of a cat and the question "What breed is the cat in the picture?" with two options: "A: Ragdoll" and "B: Silver Shaded". This data annotation task was distributed to three first annotation clients: Client 1, Client 2, and Client 3. Client 1 selected option A for this data annotation task, believing the cat in the picture is a Ragdoll, meaning Client 1's first annotation data is A; Client 2 also selected option A for this data annotation task, believing the cat in the picture is a Ragdoll, meaning Client 2's first annotation data is A; Client 3 selected option B for this data annotation task, believing the cat in the picture is a Silver Shaded, meaning Client 3's first annotation data is B.

[0066] After obtaining the first labeled data, the first labeled data can be added to the dataset to be detected.

[0067] Table 2 is an example of a dataset to be detected in the data annotation method provided in this application.

[0068]

[0069] Table 2 is the set of data to be detected obtained based on Table 1 above. Based on the first labeled data in Table 1, the set of data to be detected is (A, A, B).

[0070] Step S102: Determine whether the data in the dataset to be detected meets the preset conditions;

[0071] In this embodiment of the application, the data annotation task may include at least a first delivery stage, the first annotation client is the client delivered in the first delivery stage, and correspondingly, the first annotation data is the annotation result corresponding to the first delivery stage.

[0072] After obtaining the dataset to be tested, it can be determined whether the data in the dataset meets preset conditions. These preset conditions are the convergence conditions for the data labeling task; for example, the preset conditions could be that the data in the dataset reaches a preset confidence level.

[0073] When the set of data to be tested that contains only the first labeled data meets the preset conditions, the data labeling task does not need to go through a second delivery stage. The labeling result of the data labeling task can be determined directly based on the data in the set of data to be tested (i.e., the first labeled data).

[0074] If the data in the dataset to be detected, which only contains the first labeled data, does not meet the preset conditions, the data labeling task needs to undergo a second delivery stage, specifically the following steps S103 to S105:

[0075] Step S103: When the data set to be detected does not meet the preset conditions, a first annotation request for the data annotation task is sent to the second annotation client. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results.

[0076] The second annotation client refers to the client deployed during the second deployment phase of the data annotation task. If the data in the dataset to be detected, which only contains the first annotation data, does not meet the preset conditions, the second deployment phase can continue, deploying the data annotation task to the second annotation client.

[0077] In this embodiment of the application, when the data in the dataset to be detected does not meet the preset conditions, a first annotation request for the data annotation task can be sent to the second annotation client. The first annotation request may include a request to annotate the data annotation task and a request to parse the annotated results. In this way, after receiving the data annotation task, the second annotation client can annotate the data annotation task and parse the annotated results.

[0078] It should be noted that the first delivery stage of the data annotation task can be understood as the stage of delivering data annotation tasks not configured in parsing mode to the aforementioned multiple first annotation clients. Step S101 can be understood as: obtaining the first annotation data for the data annotation tasks not configured in parsing mode from the multiple first annotation clients respectively. Step S103 can be understood as: when the data in the dataset to be detected does not meet the preset conditions, delivering data annotation tasks configured in parsing mode to the second annotation client, so that the annotator can annotate the data annotation tasks through the second annotation client while simultaneously parsing the annotated results.

[0079] Step S104: Obtain the second annotation data for the data annotation task from the second annotation client, and the parsed content for the second annotation data, so as to display at least a portion of the parsed content in the second annotation client that is subsequently deployed;

[0080] Once each second annotation client completes its annotation, the second annotation data for that data annotation task can be obtained, along with the parsed content for that second annotation data. The second annotation data represents the annotation result of the second annotation client for the data annotation task, and the parsed content represents the reason for labeling the data task as second annotation data.

[0081] Table 3 shows an example of the second annotation data and the parsed content of the second annotation data in the data annotation method provided in this application embodiment.

[0082]

[0083] In Table 3 above, the second annotation client is client 4. Client 4 selected option A for this data annotation task, and believes that the cat in the picture is a Ragdoll cat. That is, the second annotation data corresponding to client 4 is A. The parsing content of client 4 is: the cat in the picture has dark blue eyes and is a Ragdoll cat.

[0084] It should be noted that, in this embodiment of the application, when the parsing content of the second annotation client for the second annotation data is obtained, at least a portion of the obtained parsing content can be displayed to the second annotation client during the subsequent deployment process. This allows subsequent annotators to refer to the displayed parsing content when annotating through the second annotation client, thereby assisting the subsequent second annotation client in making more accurate judgments and improving the accuracy of the annotation results. In this way, annotation data that gradually becomes more consistent and has higher accuracy can be obtained quickly, thereby achieving rapid convergence of the data annotation task.

[0085] Step S105: Add the second labeled data to the data set to be detected, and return to the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions;

[0086] In this embodiment of the application, when each second annotation client completes annotation and obtains the second annotation data for the data annotation task, the second annotation data can be added to the above-mentioned data set to be detected, thereby updating the data set to be detected.

[0087] Table 4 is an example of updating the dataset to be detected in the data annotation method provided in this application.

[0088]

[0089] Table 4 above shows how the data set to be detected is updated based on the data set to be detected exemplified in Table 2 above and the second labeled data exemplified in Table 3 above. The data set to be detected is (A, A, B). When the second labeled data "A" for the data labeling task is obtained, the second labeled data is added to the data set to be detected, thus obtaining the updated data set to be detected (A, A, B, A).

[0090] In this embodiment of the application, when the acquired second labeled data is added to the data set to be detected to obtain a new data set to be detected, the above step S102 can be returned to be executed until the data in the obtained data set to be detected meets the above preset conditions.

[0091] Step S106: Determine the annotation result of the data annotation task based on the data in the dataset to be detected.

[0092] In this embodiment of the application, when the dataset to be detected meets the above-mentioned preset conditions, the annotation result of the above-mentioned data annotation task can be determined based on the data in the dataset to be detected. Specifically, this can include the following implementation methods:

[0093] In one alternative implementation, step S106 refers to: identifying the labeled data that appears most frequently in the dataset to be detected as the labeling result of the data labeling task.

[0094] Based on Tables 2 and 4 above, this implementation method is described. Assuming that the data set to be detected (A, A, B) in Table 2 meets the above preset conditions, then the option A with the highest frequency can be used as the labeling result of the data labeling task "What breed of cat is in the picture?". Assuming that the data set to be detected (A, A, B, A) in Table 4 meets the above preset conditions, then the option A with the highest frequency can be used as the labeling result of the data labeling task "What breed of cat is in the picture?".

[0095] In another optional implementation, step S106 refers to: obtaining the weight of the annotation client corresponding to each labeled data in the dataset to be detected; and determining the annotation result of the data annotation task based on each labeled data and the corresponding weight.

[0096] In this implementation, the weight of the annotation client corresponding to each annotation data point can be recorded when adding each annotation data point to the dataset to be inspected. Thus, when the data in the dataset to be inspected meets preset conditions, the annotation result of the data annotation task can be determined based on each annotation data point in the dataset and its corresponding weight. Specifically, the weighted vote count for each annotation data point can be determined based on its weight and the number of elements in the dataset to be inspected, and the annotation data point with the highest weighted vote count is determined as the annotation result of the aforementioned data annotation task.

[0097] Optionally, the first annotation client is the first weight, the second annotation client is the second weight, the first weight is less than the second weight, and the sum of the first weight and the second weight is 1.

[0098] Using Table 4 above as an example, assuming the dataset to be tested (A, A, B, A) meets the preset conditions, with a first weight of 30% and a second weight of 70%, then there are 2 first-labeling clients selecting option A, 1 second-labeling client selecting option A, and a weighted vote count of 2*30%+1*70%=1.3. Similarly, there is 1 first-labeling client selecting option B, 0 second-labeling clients selecting option B, and a weighted vote count of 1*30%+0*70%=0.3. Thus, option A can be determined as the labeling result for the data labeling task "What breed of cat is in the picture?".

[0099] This setup assigns a lower weight to the first annotation client that annotates the data without referencing the parsed content, and a higher weight to the second annotation client that annotates the data with reference to the parsed content. Since the second annotation client with reference to the parsed content has more annotation auxiliary information, the corresponding second annotation data has a higher accuracy rate than the first annotation data. This increases the contribution of the second annotation data to the annotation results of the data annotation task, thereby improving the accuracy and reliability of the annotation results.

[0100] Optionally, each annotation client can have corresponding user profile information, and the weight of each annotation client can be determined based on the user profile information and the data annotation task. Among them, users whose user profile information and data annotation task have a strong correlation can have a higher weight.

[0101] User profiles for annotation clients refer to information created by collecting and analyzing user data from annotation clients, with their permission, to represent the typical characteristics and behavioral patterns of the annotation clients. For example, the user profile information for annotation client 1 is: part-time student, 20 years old, female, intermediate computer skills, familiar with basic office software, and skilled at annotating animal species; the user profile information for annotation client 2 is: freelancer, 30 years old, male, advanced computer skills, extensive software usage experience, and skilled at annotating sports videos.

[0102] Using Tables 1-4 above as an example, assuming the dataset to be tested (A, A, B, A) meets the preset conditions, with Client 1 having a weight of 60%, Client 2 having a weight of 70%, and Client 3 having a weight of 40% in the first annotation client, and Client 4 having a weight of 50% in the second annotation client, then the weighted votes for option A are 1*60%+1*70%+1*50%=1.8, and the weighted votes for option B are 1*40%=0.4. Thus, option A can be determined as the annotation result for the data annotation task "What breed of cat is in the picture?".

[0103] This setup determines the weight of each annotation client by the relevance of user profile information to the data annotation task. Clients with stronger relevance have higher weights, while those with weaker relevance have lower weights. This ensures that the annotation data from clients skilled in the data annotation task contributes more to the annotation results than the annotation data from clients less skilled in the data annotation task, thereby improving the accuracy and reliability of the data annotation results.

[0104] Optionally, in this embodiment of the application, historical annotation task information of each annotation client can also be obtained. The historical annotation task information includes the types of historical annotation tasks and the number of each type. The weight of each annotation client is determined based on the number of targets of the target type corresponding to the data annotation task in the historical annotation task information. The larger the number of targets, the higher the weight.

[0105] The data annotation method provided in this application includes the following steps: acquiring first annotation data from multiple first annotation clients for a data annotation task; adding the first annotation data to a dataset to be tested, and determining whether the dataset to be tested meets preset conditions; when the dataset to be tested does not meet the preset conditions, sending a first annotation request for the data annotation task to a second annotation client, the first annotation request including a request to annotate the data annotation task and a request to parse the annotated results; acquiring second annotation data from the second annotation client for the data annotation task, and parsing content of the second annotation data, so as to display at least a portion of the parsing content in the second annotation client that is subsequently deployed; adding the second annotation data to the dataset to be tested, and returning to the step of determining whether the dataset to be tested meets the preset conditions, until the dataset to be tested meets the preset conditions; and determining the annotation result of the data annotation task based on the data in the dataset to be tested.

[0106] As can be seen, the data annotation method provided in this application, when the first annotation data of the first annotation client for the data annotation task does not meet the preset conditions, requests the second annotation client to annotate while simultaneously providing an explanation of the annotation results when delivering the data annotation task to the second annotation client. Thus, upon obtaining the second annotation data and its parsed content from the second annotation client for the data annotation task, the parsed content is presented to subsequent second annotation clients. This assists subsequent second annotation clients in performing more accurate annotations using the parsed content as a reference. This allows for the rapid acquisition of increasingly consistent and highly accurate annotation data, thereby achieving rapid convergence of the data annotation task. Therefore, the data annotation method provided in this application can reduce the cost of data annotation and improve the efficiency and accuracy of annotation results.

[0107] In an optional implementation, the step S102 above, "determining whether the data set to be detected meets the preset conditions," may include the following steps S1021 and S1022:

[0108] Step S1021: Determine whether the dataset to be detected meets the preset confidence level;

[0109] Step S1022: If the data set to be detected meets the preset confidence level, determine that the data set to be detected meets the preset condition.

[0110] In this embodiment of the application, a confidence level to be achieved can be preset for the data annotation task. This preset confidence level is also known as the pre-set confidence level, for example, a pre-set confidence level of 95%. In this way, when the data set to be tested is obtained, it can be determined whether the data set to be tested meets the preset confidence level. If the data set to be tested meets the preset confidence level, it is determined that the data set to be tested meets the preset condition.

[0111] Optionally, step S1021 above can be achieved through the following steps S201 to S203:

[0112] Step S201: Determine the confidence value corresponding to the preset confidence level;

[0113] Step S202: Determine the confidence interval of the dataset to be detected based on the confidence value;

[0114] Step S203: If the width of the confidence interval is less than or equal to the preset width, determine that the data set to be detected meets the preset confidence level.

[0115] In statistics, the confidence level and corresponding confidence value refer to the critical value used when calculating a confidence interval. This value is the key numerical value used to determine the range of the confidence interval. Different confidence levels correspond to different confidence values. Confidence levels are usually expressed as a percentage, such as 90%, 95%, 99%, etc. The confidence value, on the other hand, is a specific value obtained by finding the value within the standard normal distribution (Z-distribution) or t-distribution based on the chosen confidence level. This value is used to determine the width of the confidence interval.

[0116] In practical applications, the confidence values ​​corresponding to the preset confidence levels can be obtained by querying the standard normal distribution table. The confidence value corresponding to the 90% confidence level is approximately 1.645; the confidence value corresponding to the 95% confidence level is approximately 1.96; and the confidence value corresponding to the 99% confidence level is approximately 2.576.

[0117] After obtaining the confidence value corresponding to the preset confidence level, the confidence interval of the dataset to be tested can be determined based on this confidence value. A confidence interval is an interval estimation method used in statistics to estimate population parameters. A confidence interval provides a range that contains the true value of an unknown population parameter with a certain probability (i.e., at a certain confidence level). The two endpoints of the confidence interval are called confidence limits, which are the upper and lower limits of the confidence interval, respectively.

[0118] In this embodiment, the width of the confidence interval can be preset. This preset confidence interval width is called the default width. If the width of the confidence interval calculated in step S202 is less than or equal to the default width, then the corresponding dataset to be detected is determined to meet the default confidence level. For example, if the calculated confidence interval width of the dataset to be detected is 0.5 and the default width is 0.6, then the dataset to be detected meets the default confidence level.

[0119] If the width of the confidence interval calculated in step S202 is greater than the preset width, then the corresponding dataset to be tested is determined not to meet the preset confidence level. For example, if the width of the calculated confidence interval for the dataset to be tested is 1, and the preset width is 0.6, then the dataset to be tested does not meet the preset confidence level.

[0120] In this embodiment, the confidence interval of the dataset to be tested can be determined using an interval estimation algorithm. Interval estimation is a statistical method used to estimate the possible range of an unknown parameter. This method measures the reliability of the estimate given a preset confidence level. Specifically, the interval estimation algorithm estimates an interval of the population parameter based on sample data, such that this interval contains the true parameter value with a certain probability (i.e., at the given preset confidence level).

[0121] For different types of data labeling tasks, the above interval estimation algorithm can be applied through different specific implementation methods to determine the corresponding confidence interval.

[0122] In the optional specific implementation method one, the above-mentioned data annotation task is a task of inputting numerical values, and the above-mentioned step S202 can be implemented in the following way:

[0123] Determine the mean and standard deviation of the dataset to be tested;

[0124] The confidence interval of the dataset to be tested is determined based on the confidence value, the mean, and the standard deviation.

[0125] When the data labeling task is a task that inputs numerical values, the mean and standard deviation of the dataset to be tested can be determined based on the values ​​of each labeled data in the dataset to be tested. Then, based on the confidence value, mean, standard deviation and the number of elements in the dataset to be tested, the confidence interval of the dataset to be tested can be determined.

[0126] In this embodiment of the application, when the data annotation task is a task that inputs numerical values, the confidence interval of the data set to be detected can be calculated using the following formula (1):

[0127] Formula (1)

[0128] In the above formula (1), When the data labeling task is a task with numerical input, the confidence interval of the dataset to be tested is given. The mean of the dataset to be tested. The confidence value corresponding to the preset confidence level. Let be the standard deviation of the dataset to be tested. The number of elements in the dataset to be tested.

[0129] For example, the dataset to be tested includes 100 labeled data points. The mean of the 100 labeled data points is 50 and the standard deviation is 10. Then, the confidence interval corresponding to the 95% confidence level can be calculated using the above formula (1) as [48.04, 51.96].

[0130] In the optional second implementation, the data annotation task is a task of selecting options, and step S202 can be implemented in the following way:

[0131] Determine the probability of each option in the dataset to be detected;

[0132] The confidence interval of the dataset to be detected is determined based on the confidence value and the probability of each option.

[0133] When the data labeling task is to select options, we can first calculate the probability of each option in the dataset to be tested. Then, based on the confidence value, the probability of each option, and the number of elements in the dataset to be tested, we can determine the confidence interval of the dataset to be tested.

[0134] In this embodiment of the application, when the data annotation task is an option selection task, the confidence interval of a certain option in the dataset to be detected can be calculated using the following formula (2):

[0135] Formula (2)

[0136] In the above formula (2), When the data labeling task is a selection task, the confidence interval of the dataset to be tested is given. Let be the probability of a certain option in the dataset to be tested. The confidence value corresponding to the preset confidence level. The number of elements in the dataset to be tested.

[0137] For example, the dataset to be tested includes 100 labeled data, of which 50 are option A and 50 are option B. Then, the selection ratio of option A can be calculated using the above formula (2) with a 95% confidence interval of [0.402, 0.598] and the selection ratio of option B with a 95% confidence interval of [0.402, 0.598].

[0138] By using the above method, it is determined whether the dataset to be tested meets the preset confidence level. If the confidence level is met, it means that the dataset to be tested has converged and a unique labeling result has been obtained.

[0139] In an optional implementation, the data annotation task described above may be set with a maximum number of deployments. The data annotation method provided in this application embodiment may further include the following steps:

[0140] If the dataset to be detected does not meet the preset confidence level, obtain the first number of annotation clients that have completed annotation;

[0141] If the first quantity reaches the maximum deployment quantity, it is determined that the data to be detected meets the preset conditions.

[0142] In this embodiment, the deployment of data annotation tasks can be constrained by setting a preset confidence level or a maximum deployment quantity. The maximum deployment quantity can be, for example, 50, 80, 100, etc., and can be set according to the actual situation. This application does not impose any restrictions on this.

[0143] If the width of the confidence interval calculated in step S202 is greater than the preset width, it is determined that the dataset to be tested does not meet the preset confidence level. In this case, the first number of annotation clients that have completed annotation can be obtained, for example, 10 annotation clients that have completed annotation. If the first number reaches the maximum deployment quantity, it is determined that the dataset to be tested meets the preset condition. At this time, the data annotation task is stopped, and the annotation result of the data annotation task is determined based on the data in the dataset to be tested.

[0144] In this embodiment, it is first determined whether the set to be detected meets the preset confidence level. If the preset confidence level is met, a labeling result with high accuracy and consistency is obtained, and the data labeling task can be stopped. If the preset confidence level is not met, it is determined whether the number of labeling clients that have been deployed has reached the maximum deployment number set for the data labeling task. If it has, the deployment of the data labeling task is stopped, thus solving the problem of high cost caused by deploying a large number of data labeling tasks.

[0145] In an optional implementation, step S103 can be achieved through the following steps:

[0146] Based on the set of data to be detected, determine at least a second number of data annotation tasks that need to be deployed when the preset confidence level is reached;

[0147] If the sum of the second quantity and the first quantity is greater than the maximum deployment quantity, a first annotation request for the data annotation task is sent to the second annotation client.

[0148] In this embodiment, based on the labeled data in the dataset to be detected, it can be determined at least a second number of data labeling tasks are required to reach the preset confidence level. In other words, based on the current dataset as a sample, at least a second number of data labeling tasks need to be deployed. The resulting labeled data, when added to the dataset to be detected, enables the new dataset to reach the preset confidence level.

[0149] Optionally, the step "determining, based on the set of data to be detected, at least a second number of data annotation tasks need to be deployed to reach the preset confidence level" can be achieved through the following steps:

[0150] Identify the target annotation data that appears most frequently in the dataset to be detected;

[0151] In the first state, a second quantity of data annotation tasks is required to be deployed when the preset confidence level is reached. The first state is the state in which subsequent annotation results are the target annotated data.

[0152] In this embodiment, the target annotation data with the highest frequency in the dataset to be detected can be determined first. In the first state, the annotation data corresponding to the second annotation client that is subsequently deployed are all the target annotation data, which can ensure that the aforementioned preset confidence level is met as quickly as possible after the subsequent annotation data is added to the dataset to be detected.

[0153] Taking Table 1 above as an example, in the dataset (A, A, B) to be detected in Table 1, the target labeled data with the highest frequency is A. When the labeled data of the subsequent second labeled clients are all A, it is sufficient to distribute the data to 50 people to meet the preset confidence level. Therefore, the second number is 50.

[0154] After obtaining the second quantity, if the sum of the second quantity and the first quantity of currently completed annotation clients is greater than the maximum deployment quantity mentioned above, then a first annotation request for the data annotation task is sent to the second annotation client. That is, a data annotation task configured in parsing mode is sent to the second annotation client so that while the second annotation client is annotating the data annotation task, it also inputs parsing content for the annotated data to be presented to subsequent annotators as auxiliary annotation information, forming a virtuous cycle. In this way, the data annotation task can converge quickly during deployment, improving the accuracy and efficiency of data annotation.

[0155] Table 5 is an example of a data annotation task being deployed by the second annotation client in the data annotation method provided in this application embodiment.

[0156]

[0157] In Table 5 above, the dataset to be tested is (A, A, B), meaning the number of annotation clients that have completed annotation is 3. This dataset does not meet the preset confidence level and has not reached the maximum number of submissions of 50. In this case, if the annotation data of the subsequent annotation clients is all A, it is calculated that at least 50 more annotation tasks need to be submitted to different annotation clients to meet the preset confidence level. Thus, the second number is 50, and the sum of the first and second numbers is 53, which is greater than the maximum number of submissions. Therefore, the parsing mode needs to be enabled. While submitting the data annotation task to the subsequent second annotation clients, the parsing mode is requested to input the parsing information so that the subsequent second annotation clients can input the reason for selecting the corresponding option while annotating the data annotation task.

[0158] In this setup, when the sum of the currently calculated minimum required second quantity and the already completed first quantity exceeds the maximum deployment quantity, it indicates that a large number of deployments of the data annotation task are needed to meet the preset confidence level and obtain accurate and unique annotation results. Thus, after sending a first annotation request to the second annotation client, which includes annotation and parsing of the data annotation task, the parsed content can be collected and presented to subsequent second annotation clients. This assists subsequent second annotation clients in performing more accurate annotations of the data annotation task based on the parsing content of previous annotators, thereby quickly making the annotated data more consistent and avoiding the problem of excessive costs caused by an excessive number of deployments required to meet the preset confidence level.

[0159] In an optional implementation, the data annotation method provided in this application may further include the following steps:

[0160] If the sum of the second quantity and the first quantity is less than or equal to the maximum deployment quantity, a second annotation request for the data annotation task is sent to the second annotation client. The second annotation request includes a request to annotate the data annotation task.

[0161] After obtaining the second quantity, if the sum of the second quantity and the first quantity of the currently completed annotation clients is less than or equal to the maximum deployment quantity mentioned above, a second annotation request for the data annotation task is sent to the second annotation client. The second annotation request only includes a request to annotate the data annotation task, that is, a data annotation task not configured as parsing mode is sent to the second annotation client so that the second annotation client can annotate the data annotation task.

[0162] Table 6 is an example of a data annotation task being deployed by the second annotation client in the data annotation method provided in this application embodiment.

[0163]

[0164] In Table 6 above, the data set to be tested is (A, A, B), which means that the first number of annotation clients that have completed annotation is 3. This data set to be tested does not meet the preset confidence level and has not reached the maximum number of deployments of 50. At this time, if the annotation data of the subsequent annotation clients is all A, it is calculated that at least 40 more data annotation tasks need to be deployed to different annotation clients to meet the preset confidence level. Thus, the second number is 40, and the sum of the first and second numbers is 43, which is less than the maximum number of deployments. Therefore, there is no need to enable the parsing mode. It is only necessary to deploy data annotation tasks without parsing requests to the subsequent second annotation clients, so that the subsequent second annotation clients only need to annotate the data annotation task and do not need to input parsing content.

[0165] With this setup, when the sum of the minimum required second quantity calculated and the first quantity already labeled is less than or equal to the maximum number of deployments, it means that the preset confidence level can be met within a smaller number of deployments, resulting in accurate and unique labeling results. In this way, without needing to enable the parsing mode, the second labeling client only needs to send a second labeling request that includes labeling for the data labeling task, and convergence can be quickly achieved within a smaller number of deployments.

[0166] As can be seen, the data annotation method provided in this application embodiment flexibly controls whether to configure the data annotation task to parsing mode based on the current annotation situation, thereby quickly obtaining accurate and consistent annotation data within a small number of deployments, greatly reducing the cost of data annotation, and at the same time improving the efficiency and accuracy of data annotation.

[0167] In an optional implementation, the step S103 above, "sending a first annotation request to the second annotation client for the data annotation task," can be achieved through the following steps:

[0168] For the first deployed second annotation client, a first annotation request for the data annotation task is sent to the first deployed second annotation client so that the data annotation task can be displayed through the first deployed second annotation client;

[0169] For the i-th second annotation client, a first annotation request for the data annotation task and at least part of the target parsing content are sent to the i-th second annotation client so that the data annotation task and at least part of the target parsing content can be displayed through the i-th second annotation client;

[0170] Where i iterates through 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the (i-1)th second annotation client respectively.

[0171] In this embodiment of the application, when the parsing mode is enabled, during the process of delivering a data annotation task configured in parsing mode to the second annotation client, a first annotation request for the data annotation task can be sent to the first delivered second annotation client. In the graphical user interface of the first delivered second annotation client, the annotation area of ​​the data annotation task can be displayed, and a parsing input area can be provided at the same time, so that the annotator can annotate the data annotation area in the annotation area through the first delivered second annotation client, and input the parsing content in the parsing input area.

[0172] For any i-th second annotation client among the second to Nth second annotation clients, a first annotation request for the data annotation task and at least a portion of the parsed content corresponding to the first to (i-1)th second annotation clients can be sent to that i-th second annotation client. Thus, the graphical user interface of the i-th second annotation client can display the annotation area for the data annotation task and simultaneously provide a parsing input area. Furthermore, the at least portion of the target parsing content can be displayed in the annotation area, specifically presented in a bullet-screen format.

[0173] In one alternative implementation, at least a portion of the target parsing content can be displayed in the second annotation client that is the i-th deployment, using a display strategy that differentiates the parsing content corresponding to different labeled data. For example, if the data annotation task is an option-selection task, including option A and option B, the parsing content corresponding to option A can be displayed in red and the parsing content corresponding to option B can be displayed in blue in the second annotation client that is the i-th deployment.

[0174] It should be noted that this application is not limited to the display strategy of using different colors to distinguish the display of the parsing content corresponding to different annotation data. It can also be the display strategy of using different fonts to distinguish the display of the parsing content corresponding to different annotation data, or the display strategy of using different display positions to distinguish the display of the parsing content corresponding to different annotation data. This application does not limit this.

[0175] In this way, the parsed content entered by the previous client can be intuitively displayed to the subsequent second annotation client, and the display strategy for the parsed content corresponding to different annotation data can be distinguished. This provides visual guidance to subsequent annotators, enabling them to obtain important information in the parsed content corresponding to different annotation data in a timely manner. Furthermore, annotators can clearly understand the selection situation corresponding to each annotation data, which facilitates users to make quick decisions and further improves the efficiency of data annotation.

[0176] The following, combined with Figure 3 The complete process of the data annotation method provided in the first embodiment of this application is described, including the following steps S301 to S311:

[0177] Step S301: Send a second annotation request for the data annotation task to multiple first annotation clients;

[0178] Step S302 (i.e.) Figure 2 Step S101): Obtain first annotation data from multiple first annotation clients for the data annotation task, and add the first annotation data to the data set to be detected;

[0179] Step S303 (also known as the aforementioned step S1021): Determine whether the current dataset to be detected meets the preset confidence level;

[0180] If the result of step S303 is negative, proceed to step S304: determine whether the current number of completed annotations has reached the maximum deployment quantity.

[0181] If the result of step S304 is negative, proceed to step S305: determine the minimum second quantity required to be deployed when the preset credit level is met;

[0182] Step S306: Determine whether the sum of the second quantity and the currently marked first quantity is greater than the maximum deployment quantity;

[0183] If the result of step S306 is yes, proceed to step S307: send the first annotation request for the data annotation task to the second annotation client;

[0184] Step S308 (i.e., the aforementioned steps S104 and S105): Obtain the second annotation data and the parsing content of the second annotation data from the second annotation client for the data annotation task, so as to present the parsing content to the subsequent second annotation client, add the second annotation data to the data set to be detected, and return to execute step S303;

[0185] If the result of step S306 is yes, proceed to step S309: send a second annotation request for the data annotation task to the second annotation client;

[0186] Step S310: Obtain the second annotation data for the data annotation task from the second annotation client, add the second annotation data to the data set to be detected, and return to execute step S303;

[0187] If the result of step S303 is yes, proceed to step S311: determine the annotation result of the data annotation task based on the data in the dataset to be detected;

[0188] If the result of step S304 is yes, then execute step S311 above.

[0189] Steps S301 to S311 described above can be referred to the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0190] In this embodiment, each time the dataset to be tested is updated, it can be first determined whether the dataset meets a preset confidence level. If the preset confidence level is met, delivery is stopped, and the annotation result of the data annotation task is determined based on the data in the dataset to be tested. If the preset confidence level is not met, it is determined whether the first number of annotation clients that have completed annotation has reached the maximum delivery number. If the maximum delivery number has been reached, delivery is stopped, and the annotation result of the data annotation task is determined based on the data in the dataset to be tested. When the maximum deployment quantity is not reached, a second quantity of data annotation tasks is calculated to meet the preset confidence level. If the sum of the second and first quantities exceeds the maximum deployment quantity, parsing mode is activated, and a first annotation request for the data annotation task is sent to the second annotation client. This first annotation request includes requests to annotate the data annotation task and to parse the annotated data. This allows subsequent annotation clients to annotate the data annotation task while simultaneously inputting parsed content into the annotated data. The input parsed content is then displayed to subsequent annotation clients, enabling them to refer to the parsed content from previous annotation clients and make correct judgments. This quickly brings the annotated data to consistency, avoiding the problem of excessive costs caused by an excessive deployment quantity required to meet the preset confidence level. When the sum of the second and first quantities is less than or equal to the maximum deployment quantity, a second annotation request for the data annotation task is sent to the second annotation client. This second annotation request includes requests to annotate the data annotation task without requiring parsing the annotated data. This allows subsequent annotation clients to annotate the data annotation task.

[0191] The second embodiment of this application provides a display method, such as... Figure 4 The diagram shown is a flowchart of the display method provided in the second embodiment of this application, including steps S401 to S403:

[0192] Step S401: In response to receiving a first annotation request for a data annotation task, the annotation area and parsing input area of ​​the data annotation task are displayed through a second annotation client. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results.

[0193] Step S402: In response to the annotation operation in the annotation area, obtain the second annotation data for the data annotation task;

[0194] Step S403: In response to an input operation in the parsing input area, obtain the parsing content for the second annotation data input, so as to display at least a portion of the parsing content in the second annotation client that is subsequently deployed.

[0195] The above display method can be applied to a second annotation client. When the second annotation client receives a first annotation request that includes annotating a data annotation task and parsing the annotated results, the annotation interface of the data annotation task can be displayed through the graphical user interface of the second annotation client. The annotation interface includes an annotation area for annotating the data annotation task and a parsing input area for parsing the annotated results.

[0196] Annotators holding the second annotation client can perform annotation operations in the annotation area. Upon receiving such an annotation operation, they can obtain the second annotation data for that data annotation task. Furthermore, annotators holding the second annotation client can perform input operations in the parsing input area. Upon receiving such an input operation, they can obtain the parsed content for the second annotation data input, so that at least a portion of the parsed content can be displayed in the subsequently deployed second annotation client.

[0197] Optionally, the parsing input area may include at least one of a preset parsing options area and a custom parsing content area. The preset parsing content area includes multiple optional preset parsing options, from which the user can select a target parsing option, which is the entered parsing content. The custom parsing content area is used to support user input of custom parsing content.

[0198] The aforementioned preset parsing options can be a preset number of parsing contents that appear frequently in the currently obtained parsing content. For example, the preset parsing options are the top 10 parsing contents that appear frequently in the currently obtained parsing content.

[0199] The display method provided in the second embodiment of this application includes the following steps: in response to receiving a first annotation request for a data annotation task, displaying the annotation area and parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results; in response to an annotation operation in the annotation area, obtaining second annotation data for the data annotation task; and in response to an input operation in the parsing input area, obtaining parsing content input for the second annotation data, so as to display at least a portion of the parsing content in the subsequently deployed second annotation client. It can be seen that the display method provided in this embodiment, upon receiving a first annotation request for a data annotation task, can annotate the data annotation task and input the parsing corresponding to the annotated results through the second annotation client by displaying the annotation area and parsing input area of ​​the data annotation task in the second annotation client, and display at least a portion of the parsing content in the subsequently deployed second annotation client. Thus, by displaying the parsing content in the subsequent second annotation client, more accurate data annotation can be assisted in the subsequent second annotation client, thereby improving the efficiency of data annotation.

[0200] In an optional implementation, the display method provided in this application embodiment may further include the following steps:

[0201] In response to receiving target parsing content for the data annotation task from other annotation clients, at least a portion of the target parsing content is displayed through the second annotation client.

[0202] Optionally, the annotation area includes a question display area, and the above step "displaying at least a portion of the target parsing content through the second annotation client" can be achieved through the following steps:

[0203] At least a portion of the target analysis content is displayed in the title display area in the form of bullet comments.

[0204] Optionally, the above step "displaying at least a portion of the target parsed content through the second annotation client" can be achieved through the following steps:

[0205] The second annotation client uses a strategy to differentiate the display of parsed content corresponding to different annotation data, displaying at least a portion of the target parsed content.

[0206] In the second embodiment of this application, when the second annotation client receives the target parsing content for the data annotation task from other annotation clients, at least a portion of the target parsing content can be displayed in the second annotation client.

[0207] In the specific implementation, the target parsing content includes multiple different first contents. For a first content, the third quantity of the first content in the target parsing content can be obtained, and the first content and the corresponding third quantity can be displayed in the above-mentioned question display area in the form of bullet comments.

[0208] In this embodiment, when the second annotation client receives target parsing content for the data annotation task from other annotation clients, it can display at least a portion of the target parsing content in the second annotation client using a display strategy that differentiates the parsing content corresponding to different annotated data. This part has been described in detail in the above description of the first embodiment and will not be repeated here.

[0209] like Figure 5 The diagram shown is an example of the interface of the second annotation client in the display method provided in the second embodiment of this application. In response to receiving a first annotation request for a data annotation task, the annotation area 50 and the parsing input area 51 are displayed through the second annotation client. The annotation area 50 includes a question display area 52, in which the parsing content previously input by the second annotation client is displayed in the form of bullet comments. These include 5 bullet comments for option A "Ragdoll" such as "Ragdoll cats are large, long, and muscular", 3 bullet comments for option B "Silver Shaded" such as "Unique red nose, definitely a Silver Shaded", and 8 bullet comments for option A "Ragdoll" such as "Ragdoll has deep blue eyes". The parsing input area 51 can include a preset parsing content area and a custom parsing content area. The preset parsing content area includes multiple preset parsing contents, namely "Ragdoll cat is large", "Ragdoll cat has a broad chest", "muscular", "short and thick neck", and "dark blue eyes". The annotator can select option A in the annotation area 51 to annotate the data annotation task, and select the preset parsing contents such as "muscular" and "dark blue eyes" in the preset parsing content area in the parsing input area 51 to input the parsing content for the selected option A.

[0210] Corresponding to the data annotation method provided in the first embodiment of this application, the third embodiment of this application also provides a data annotation device, such as... Figure 6 As shown, the data annotation device 600 includes:

[0211] The first acquisition unit 601 is used to acquire the first annotation data of multiple first annotation clients for the data annotation task, and add the first annotation data to the data set to be detected;

[0212] The first determining unit 602 is used to determine whether the data set to be detected meets preset conditions;

[0213] The sending unit 603 is configured to send a first annotation request for the data annotation task to the second annotation client when the data set to be detected does not meet the preset conditions. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results.

[0214] The second acquisition unit 604 is used to acquire the second annotation data of the second annotation client for the data annotation task, and the parsing content of the second annotation data, so as to display at least a portion of the parsing content in the second annotation client that is subsequently deployed;

[0215] Adding unit 605 is used to add the second labeled data to the data set to be detected, and return to execute the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions;

[0216] The second determining unit 606 is used to determine the annotation result of the data annotation task based on the data in the dataset to be detected.

[0217] Optionally, the first determining unit 602 is specifically used for:

[0218] Determine whether the dataset to be tested meets the preset confidence level;

[0219] If the dataset to be tested meets the preset confidence level, it is determined that the dataset to be tested meets the preset condition.

[0220] Optionally, the data labeling task is set with a maximum number of deployments, and the first determining unit 602 is further specifically used for:

[0221] If the dataset to be detected does not meet the preset confidence level, obtain the first number of annotation clients that have completed annotation;

[0222] If the first quantity reaches the maximum deployment quantity, it is determined that the data set to be detected meets the preset condition.

[0223] Optionally, the sending unit 603 is specifically used for:

[0224] Based on the set of data to be detected, determine at least a second number of data annotation tasks that need to be deployed when the preset confidence level is reached;

[0225] If the sum of the second quantity and the first quantity is greater than the maximum deployment quantity, a first annotation request for the data annotation task is sent to the second annotation client.

[0226] Optionally, the transmitting unit 603 is further specifically used for:

[0227] If the sum of the second quantity and the first quantity is less than or equal to the maximum deployment quantity, a second annotation request for the data annotation task is sent to the second annotation client. The second annotation request includes a request to annotate the data annotation task.

[0228] Optionally, the sending unit 603 is specifically used for:

[0229] For the first deployed second annotation client, a first annotation request for the data annotation task is sent to the first deployed second annotation client so that the data annotation task can be displayed through the first deployed second annotation client;

[0230] For the i-th second annotation client, a first annotation request for the data annotation task and at least part of the target parsing content are sent to the i-th second annotation client so that the data annotation task and at least part of the target parsing content can be displayed through the i-th second annotation client;

[0231] Where i iterates through 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the (i-1)th second annotation client respectively.

[0232] Optionally, the sending unit 603 is specifically used for:

[0233] Identify the target annotation data that appears most frequently in the dataset to be detected;

[0234] In the first state, a second quantity of data annotation tasks is required to be deployed when the preset confidence level is reached. The first state is the state where subsequent annotation data is the target annotation data.

[0235] Optionally, the first determining unit 602 is specifically used for:

[0236] Determine the confidence value corresponding to the preset confidence level;

[0237] Based on the confidence value, determine the confidence interval of the dataset to be detected;

[0238] If the width of the confidence interval is less than or equal to the preset width, the data set to be detected is determined to meet the preset confidence level.

[0239] Optionally, the data labeling task is a task of inputting numerical values, and the first determining unit 602 is specifically used for:

[0240] Determine the mean and standard deviation of the dataset to be tested;

[0241] The confidence interval of the dataset to be tested is determined based on the confidence value, the mean, and the standard deviation.

[0242] Optionally, the data annotation task is a task of selecting options, and the first determining unit 602 is specifically used for:

[0243] Determine the probability of each option in the dataset to be detected;

[0244] The confidence interval of the dataset to be detected is determined based on the confidence value and the probability of each option.

[0245] Optionally, the second determining unit 606 is specifically used for:

[0246] Obtain the weight of the annotation client corresponding to each labeled data in the dataset to be detected;

[0247] The annotation result of the data annotation task is determined based on the labeled data and the corresponding weights.

[0248] Corresponding to the display method provided in the second embodiment of this application, the fourth embodiment of this application also provides a display device, such as... Figure 7 As shown, the display device 700 includes:

[0249] The first display unit 701 is configured to, in response to receiving a first annotation request for a data annotation task, display the annotation area and parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results;

[0250] The third acquisition unit 702 is configured to acquire second annotation data for the data annotation task in response to an annotation operation in the annotation area; and to acquire parsing content input for the second annotation data in response to an input operation in the parsing input area, so as to display at least a portion of the parsing content in a subsequently deployed second annotation client.

[0251] Optionally, the first display unit 701 is further configured to:

[0252] In response to receiving target parsing content for the data annotation task from other annotation clients, at least a portion of the target parsing content is displayed through the second annotation client.

[0253] Optionally, the annotation area includes a question display area, and the first display unit 701 is further specifically used for:

[0254] At least a portion of the target analysis content is displayed in the title display area in the form of bullet comments.

[0255] Optionally, the first display unit 701 is specifically used for:

[0256] The second annotation client uses a strategy to differentiate the display of parsed content corresponding to different annotation data, displaying at least a portion of the target parsed content.

[0257] Corresponding to the data annotation method provided in the first embodiment of this application or the display method provided in the second embodiment of this application, the fifth embodiment of this application also provides an electronic device for data annotation or display.

[0258] like Figure 8 The diagram shown is a structural block diagram of an example of an electronic device for data annotation or display provided in an embodiment of this application.

[0259] In this embodiment, an optional hardware structure of the electronic device 800 may be as follows: Figure 8 As shown, it includes: at least one processor 801, at least one memory 802 and at least one communication bus 805; the memory 802 contains a program 803 and data 804.

[0260] Bus 805 can be a communication device for transmitting data between components within electronic device 800, such as an internal bus (e.g., CPU-memory bus, where the processor is the central processing unit, or CPU for short) or an external bus (e.g., a universal serial bus port or a peripheral component interconnection fast port).

[0261] Additionally, the electronic device also includes at least one network interface 806 and at least one peripheral interface 807. The network interface 806 provides wired or wireless communication with an external network 808 (e.g., the Internet, intranet, local area network, mobile communication network, etc.). In some embodiments, the network interface 806 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (NFC) adapters, cellular network chips, etc.

[0262] Peripheral interface 807 is used to connect to peripherals, such as peripheral 1 in the figure. Figure 8 809 in the middle), peripheral 2 ( Figure 8 810 in the middle) and peripheral 3 (Figure 8 (811 in the original text). Peripherals are peripheral devices, which may include, but are not limited to, cursor control devices (such as mice, touchpads, or touchscreens), keyboards, displays (such as cathode ray tube displays, liquid crystal displays), monitors or light-emitting diode displays, video input devices (such as cameras or input interfaces coupled to video files), etc.

[0263] The processor 801 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0264] The memory 802 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage device.

[0265] The processor 801 calls the program and data stored in the memory 802 and executes the following steps:

[0266] Obtain first annotation data from multiple first annotation clients for the data annotation task, and add the first annotation data to the dataset to be detected;

[0267] Determine whether the dataset to be detected meets the preset conditions;

[0268] When the set of data to be detected does not meet the preset conditions, a first annotation request for the data annotation task is sent to the second annotation client. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results.

[0269] Obtain the second annotation data for the data annotation task from the second annotation client, and the parsed content for the second annotation data, so as to display at least a portion of the parsed content in the second annotation client that is subsequently deployed;

[0270] Add the second labeled data to the data set to be detected, and return to the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions;

[0271] Based on the data in the dataset to be detected, determine the annotation results of the data annotation task;

[0272] or

[0273] In response to receiving a first annotation request for a data annotation task, the annotation area and parsing input area of ​​the data annotation task are displayed through a second annotation client. The first annotation request includes requests to annotate the data annotation task and to parse the annotated results.

[0274] In response to a labeling operation in the labeled area, second labeled data for the data labeling task is obtained; and

[0275] In response to an input operation in the parsing input area, parsing content for the second annotation data input is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently deployed.

[0276] Compared with the data annotation method provided in the first embodiment of this application or the display method provided in the second embodiment of this application, the sixth embodiment of this application provides a computer-readable storage medium storing a program of the data annotation method provided in the first embodiment of this application or the display method provided in the second embodiment of this application. The program is executed by a processor to perform the following steps:

[0277] Obtain first annotation data from multiple first annotation clients for the data annotation task, and add the first annotation data to the dataset to be detected;

[0278] Determine whether the dataset to be detected meets the preset conditions;

[0279] When the set of data to be detected does not meet the preset conditions, a first annotation request for the data annotation task is sent to the second annotation client. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results.

[0280] Obtain the second annotation data for the data annotation task from the second annotation client, and the parsed content for the second annotation data, so as to display at least a portion of the parsed content in the second annotation client that is subsequently deployed;

[0281] Add the second labeled data to the data set to be detected, and return to the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions;

[0282] Based on the data in the dataset to be detected, determine the annotation results of the data annotation task;

[0283] or

[0284] In response to receiving a first annotation request for a data annotation task, the annotation area and parsing input area of ​​the data annotation task are displayed through a second annotation client. The first annotation request includes requests to annotate the data annotation task and to parse the annotated results.

[0285] In response to a labeling operation in the labeled area, second labeled data for the data labeling task is obtained; and

[0286] In response to an input operation in the parsing input area, parsing content for the second annotation data input is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently deployed.

[0287] It should be noted that for detailed descriptions of the display method, data annotation device, display device, electronic device, and computer-readable storage medium provided in the second, third, fourth, fifth, and sixth embodiments of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0288] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0289] In a typical configuration, a node device in a blockchain includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0290] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0291] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage media, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0292] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0293] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A data annotation method, characterized in that, The method includes: Obtain first annotation data from multiple first annotation clients for the data annotation task, and add the first annotation data to the dataset to be detected; Determine whether the dataset to be detected meets the preset conditions; When the set of data to be detected does not meet the preset conditions, a first annotation request for the data annotation task is sent to the second annotation client. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results. Obtain the second annotation data for the data annotation task from the second annotation client, and the parsed content for the second annotation data, so as to display at least a portion of the parsed content in the second annotation client that is subsequently deployed; Add the second labeled data to the data set to be detected, and return to the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions; Based on the data in the dataset to be detected, determine the annotation results of the data annotation task; Determining whether the dataset to be detected meets preset conditions includes: Determine whether the dataset to be tested meets the preset confidence level; If the dataset to be tested meets the preset confidence level, it is determined that the dataset to be tested meets the preset condition.

2. The method according to claim 1, characterized in that, The data labeling task has a maximum number of deliverables, and the method further includes: If the dataset to be detected does not meet the preset confidence level, obtain the first number of annotation clients that have completed annotation; If the first quantity reaches the maximum deployment quantity, it is determined that the data set to be detected meets the preset condition.

3. The method according to claim 2, characterized in that, When the set of data to be detected does not meet the preset conditions, sending a first annotation request for the data annotation task to the second annotation client includes: Based on the set of data to be detected, determine at least a second number of data annotation tasks that need to be deployed when the preset confidence level is reached; If the sum of the second quantity and the first quantity is greater than the maximum deployment quantity, a first annotation request for the data annotation task is sent to the second annotation client.

4. The method according to claim 3, characterized in that, The method further includes: If the sum of the second quantity and the first quantity is less than or equal to the maximum deployment quantity, a second annotation request for the data annotation task is sent to the second annotation client. The second annotation request includes a request to annotate the data annotation task.

5. The method according to claim 1, characterized in that, Sending a first annotation request for the data annotation task to the second annotation client includes: For the first deployed second annotation client, a first annotation request for the data annotation task is sent to the first deployed second annotation client so that the data annotation task can be displayed through the first deployed second annotation client; For the i-th second annotation client, a first annotation request for the data annotation task and at least part of the target parsing content are sent to the i-th second annotation client so that the data annotation task and at least part of the target parsing content can be displayed through the i-th second annotation client; Where i iterates through 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the (i-1)th second annotation client respectively.

6. The method according to claim 3, characterized in that, The step of determining, based on the set of data to be detected, at least a second number of data annotation tasks need to be deployed to reach the preset confidence level includes: Identify the target annotation data that appears most frequently in the dataset to be detected; In the first state, a second quantity of data annotation tasks is required to be deployed when the preset confidence level is reached. The first state is the state where subsequent annotation data is the target annotation data.

7. The method according to claim 1, characterized in that, Determining whether the dataset to be detected meets the preset confidence level includes: Determine the confidence value corresponding to the preset confidence level; Based on the confidence value, determine the confidence interval of the dataset to be detected; If the width of the confidence interval is less than or equal to the preset width, the data set to be detected is determined to meet the preset confidence level.

8. The method according to claim 7, characterized in that, The data annotation task is a task that inputs numerical values. The step of determining the confidence interval of the dataset to be detected based on the confidence value includes: Determine the mean and standard deviation of the dataset to be tested; The confidence interval of the dataset to be tested is determined based on the confidence value, the mean, and the standard deviation.

9. The method according to claim 7, characterized in that, The data annotation task is a task involving selecting options. Determining the confidence interval of the dataset to be detected based on the confidence value includes: Determine the probability of each option in the dataset to be detected; The confidence interval of the dataset to be detected is determined based on the confidence value and the probability of each option.

10. The method according to claim 1, characterized in that, The step of determining the annotation result of the data annotation task based on the data in the dataset to be detected includes: Obtain the weight of the annotation client corresponding to each labeled data in the dataset to be detected; The annotation result of the data annotation task is determined based on the labeled data and the corresponding weights.

11. A display method, characterized in that, The method includes: In response to receiving a first annotation request for a data annotation task, the annotation area and parsing input area of ​​the data annotation task are displayed through a second annotation client. The first annotation request includes requests to annotate the data annotation task and to parse the annotated results. In response to a labeling operation in the labeled area, second labeled data for the data labeling task is obtained; and In response to an input operation in the parsing input area, parsing content for the second annotation data input is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently deployed.

12. The method according to claim 11, characterized in that, The method further includes: In response to receiving target parsing content for the data annotation task from other annotation clients, at least a portion of the target parsing content is displayed through the second annotation client.

13. The method according to claim 12, characterized in that, The annotation area includes a question display area, and the display of at least a portion of the target parsing content through the second annotation client includes: At least a portion of the target analysis content is displayed in the title display area in the form of bullet comments.

14. The method according to claim 12, characterized in that, The step of displaying at least a portion of the target parsed content through the second annotation client includes: The second annotation client uses a strategy to differentiate the display of parsed content corresponding to different annotation data, displaying at least a portion of the target parsed content.

15. A data annotation device, characterized in that, The device includes: The first acquisition unit is used to acquire the first annotation data of multiple first annotation clients for the data annotation task, and add the first annotation data to the data set to be detected; The first determining unit is used to determine whether the data set to be detected meets preset conditions; The sending unit is configured to send a first annotation request for the data annotation task to the second annotation client when the data set to be detected does not meet the preset conditions. The first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results. The second acquisition unit is used to acquire the second annotation data of the second annotation client for the data annotation task, and the parsing content of the second annotation data, so as to display at least a portion of the parsing content in the second annotation client that is subsequently deployed; An adding unit is used to add the second labeled data to the data set to be detected, and return to the step of determining whether the data set to be detected meets the preset conditions, until the data set to be detected meets the preset conditions; The second determining unit is used to determine the annotation result of the data annotation task based on the data in the dataset to be detected; Determining whether the dataset to be detected meets preset conditions includes: Determine whether the dataset to be tested meets the preset confidence level; If the dataset to be tested meets the preset confidence level, it is determined that the dataset to be tested meets the preset condition.

16. A display device, characterized in that, The device includes: The first display unit is configured to, in response to receiving a first annotation request for a data annotation task, display the annotation area and parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and a request to parse the annotated results; The third acquisition unit is configured to acquire second annotation data for the data annotation task in response to an annotation operation in the annotation area; and to acquire parsed content input for the second annotation data in response to an input operation in the parsing input area, so as to display at least a portion of the parsed content in a subsequently deployed second annotation client.

17. An electronic device, characterized in that, include: processor; as well as A memory for storing a data processing program, which, when powered on and run by the processor, executes the method as described in any one of claims 1 to 10 or 11 to 14.

18. A computer-readable storage medium, characterized in that, The system contains a data processing program that is executed by a processor to perform the method as described in any one of claims 1-10 or 11-14.

Images