Data annotation method and device, data display method and device and electronic equipment

Abstract

The invention discloses a data annotation method and device, a display method and device, electronic equipment and a computer readable storage medium, and the data annotation method comprises the steps: obtaining first annotation data of a first annotation client for a data annotation task, and adding the first annotation data to a to-be-detected data set; if the to-be-detected data set does not meet the preset condition, a first labeling request is sent to a second labeling client, and the first labeling request comprises a request for labeling and analyzing the data labeling task; obtaining second annotation data of a second annotation client for the data annotation task and analysis content for the second annotation data, so as to display at least part of the analysis content in the second annotation client which is put later; adding the second annotation data to the to-be-detected data set until the to-be-detected data set meets a preset condition; and determining a labeling result of the data labeling task according to the to-be-detected data set. According to the method, the data annotation cost can be reduced, and the efficiency and precision of the annotation result can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data labeling method, a display method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Data labeling refers to the process of adding labels to each sample in a data set. These labels can be used to train machine learning models so that they can understand the content of the data. At present, in order to ensure the accuracy of the labeling results, the common method on the market is for multiple labelers to independently label the same data, and then determine the final labeling results through voting or expert review. However, due to factors such as differences in personal judgment and different levels of professional knowledge, current data labeling tasks often require a large number of labelers to obtain more consistent labeling results, and the accuracy of the labeling results is also low, and the efficiency is extremely low. Summary of the invention

[0003] The present application provides a data annotation method, display method, device, electronic device and computer-readable storage medium, which can reduce the cost of data annotation and improve the efficiency and accuracy of annotation results. The specific scheme is as follows:

[0004] In a first aspect, an embodiment of the present application provides a data labeling method, the method comprising:

[0005] Obtain first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected;

[0006] Determine whether the data set to be detected meets a preset condition;

[0007] When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0008] Acquire the second annotation data for the data annotation task and the parsed content for the second annotation data by the second annotation client, so as to display at least part of the parsed content in the second annotation client that is subsequently launched;

[0009] Add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition;

[0010] Determine the labeling result of the data labeling task according to the data in the to-be-detected data set.

[0011] In a second aspect, an embodiment of the present application provides a display method, the method comprising:

[0012] In response to receiving a first annotation request for a data annotation task, displaying an annotation area and a parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and to parse the annotated result;

[0013] In response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task; and

[0014] In response to an input operation in the parsing input area, parsing content inputted for the second annotation data is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently delivered.

[0015] In a third aspect, an embodiment of the present application provides a data labeling device, the device comprising:

[0016] A first acquisition unit, configured to acquire first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected;

[0017] A first determining unit, used to determine whether the to-be-detected data set meets a preset condition;

[0018] A sending unit, configured to send a first annotation request for the data annotation task to a second annotation client when the to-be-detected data set does not satisfy a preset condition, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0019] A second acquisition unit, configured to acquire second annotation data for the data annotation task and parsed content for the second annotation data from the second annotation client, so as to display at least part of the parsed content in a second annotation client that is subsequently launched;

[0020] an adding unit, configured to add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition;

[0021] The second determining unit is used to determine the labeling result of the data labeling task according to the data in the data set to be detected.

[0022] In a fourth aspect, an embodiment of the present application provides a display device, the device comprising:

[0023] A first display unit, configured to display a labeling area and a parsing input area of ​​the data labeling task through a second labeling client in response to receiving a first labeling request for the data labeling task, wherein the first labeling request includes a request for labeling the data labeling task and for parsing the labeled result;

[0024] A third acquisition unit is used to obtain second annotation data for the data annotation task in response to an annotation operation in the annotation area; and to obtain parsed content input for the second annotation data in response to an input operation in the parsing input area, so as to display at least part of the parsed content in a second annotation client that is subsequently launched.

[0025] In a fifth aspect, the present application further provides an electronic device, including:

[0026] Processor; and

[0027] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method described in the first aspect or the second aspect is executed.

[0028] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the method described in the first aspect or the second aspect.

[0029] Compared with the prior art, this application has the following advantages:

[0030] The data annotation method provided in an embodiment of the present application includes the following steps: obtaining first annotation data for a data annotation task respectively from multiple first annotation clients; adding the first annotation data to a data set to be detected, and determining whether the data set to be detected meets a preset condition; when the data set to be detected does not meet the preset condition, sending a first annotation request for the data annotation task to a second annotation client, the first annotation request including a request for annotation of the data annotation task and a request for parsing the annotated result; obtaining second annotation data for the data annotation task from the second annotation client, and parsed content for the second annotation data, so as to display at least part of the parsed content in a second annotation client that is subsequently launched; adding the second annotation data to the data set to be detected, and returning to execute the step of determining whether the data set to be detected meets the preset condition until the data set to be detected meets the preset condition; determining the annotation result of the data annotation task based on the data in the data set to be detected.

[0031] It can be seen that the data annotation method provided in the embodiment of the present application, when the first annotation data of the data annotation task in the first annotation client does not meet the preset conditions, then when the data annotation task is delivered to the second annotation client, the second annotation client is requested to annotate and the annotation result is analyzed and explained. In this way, when the second annotation data for the data annotation task and the analysis content of the second annotation data are obtained from the second annotation client, the analysis content is presented to the subsequent second annotation client, thereby assisting the subsequent second annotation client to use the analysis content as a reference for more accurate annotation. In this way, annotation data that gradually tends to be consistent and has a higher degree of accuracy can be quickly obtained, thereby achieving rapid convergence of the data annotation task. Therefore, the data annotation method provided in the embodiment of the present application can reduce the cost of data annotation and improve the efficiency and accuracy of the annotation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A diagram of a labeling system for implementing a data labeling method provided in an embodiment of the present application;

[0033] Figure 2 is a flow chart of the data labeling method provided in the first embodiment of the present application;

[0034] Figure 3 It is a schematic diagram of the entire process of the data labeling method provided in the first embodiment of the present application;

[0035] Figure 4 is a flow chart of a display method provided in the second embodiment of the present application;

[0036] Figure 5 is a schematic diagram of an interface of an example of a second annotation client in the display method provided in the second embodiment of the present application;

[0037] Figure 6 is a structural block diagram of an example of a data labeling device provided in an embodiment of the present application;

[0038] Figure 7 is a structural block diagram of an example of a display device provided in an embodiment of the present application;

[0039] Figure 8 It is a structural block diagram of an example of an electronic device for data processing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0041] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0043] It should be understood that in the embodiments of the present application, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0044] Before describing the implementation methods of the present application in detail, the prior art is first further described.

[0045] In the field of machine learning and artificial intelligence, high-quality data annotation is the basis for building accurate and reliable machine learning models. Data annotation refers to adding meaningful labels or annotations to raw data (such as images, audio, text, etc.) to help algorithms understand key information in the data. With the rapid development of artificial intelligence, the importance of data annotation has become increasingly prominent.

[0046] At present, the traditional data annotation method is to distribute data annotation tasks to a large number of scattered annotators through a data annotation platform to improve the diversity and coverage of data annotation. Usually, in order to ensure the accuracy of the annotation results, multiple annotators will independently annotate the same data set, and then decide the final annotation results through a majority voting mechanism or by expert review.

[0047] However, when it comes to data labeling tasks that are highly subjective or require professional knowledge, traditional data labeling methods face many challenges. On the one hand, even if a large number of independent labeling tasks are assigned to different labelers, it is still difficult to obtain a unique and definite result with high confidence due to factors such as differences in personal judgment and different levels of professional knowledge. This greatly reduces the efficiency of data labeling and increases the cost of data labeling. On the other hand, traditional data labeling methods mainly rely on simple majority voting or manual review, which makes it difficult to handle more complex or highly professional data labeling tasks.

[0048] Therefore, how to reduce the cost of data annotation and improve the efficiency and accuracy of data annotation has become an urgent problem to be solved.

[0049] Based on the above reasons, in order to reduce the cost of data labeling and improve the efficiency and accuracy of data labeling, the first embodiment of the present application provides a data labeling method, which is applied to electronic devices. The electronic devices can be desktop computers, laptop computers, mobile phones, tablet computers, electronic watches, etc., or other electronic devices that can be labeled, which are not specifically limited in the embodiments of the present application.

[0050] For example, in combination with the above introduction, Figure 1 A labeling system 100 for implementing a data labeling method provided by an embodiment of the present application is shown, and the labeling system 100 may include at least one labeling client 101, at least one server 102, and a network. The labeler may hold the labeling client 101 and connect to different servers 102 through the network. The labeling client runs on any device with computing hardware that can support and execute corresponding software application tools.

[0051] In the above-mentioned annotation system 100, the annotation client 101 may be an application installed with support for annotating data annotation tasks. In some cases, the annotation client 101 may be a client such as a web page or a small program that can access data annotation tasks. During the data annotation process, data can be exchanged between the annotation client 101 and the server 102.

[0052] In a possible application scenario, different annotation clients 101 may be served by different servers 102 , and the servers 102 corresponding to different annotation clients 101 may be the same server.

[0053] In addition, when the annotation system 100 includes multiple annotation clients, multiple servers, and multiple networks, different annotation clients can be connected to different servers through different networks.

[0054] The network may be a wireless network or a wired network, such as a wireless network such as a wireless local area network (WLAN), a local area network (LAN), a cellular network, a 2G network, a 3G network, a 4G network, a 5G network, etc. In addition, different annotation clients may also use their own Bluetooth network or hotspot network to connect to other annotation clients or to a server, etc. In addition, the annotation system 100 may include multiple databases, which are coupled to different servers, and data related to data annotation may be stored in the databases.

[0055] It should be noted that Figure 1 The annotation system diagram shown is merely an example. The annotation system 100 described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application.

[0056] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] The following, combined Figure 2 and Figure 3 The data labeling method provided in the first embodiment of the present application is introduced.

[0058] like Figure 2 As shown, the data labeling method provided in the first embodiment of the present application includes the following steps S101 to S106.

[0059] Step S101: obtaining first annotation data for data annotation tasks from a plurality of first annotation clients respectively, and adding the first annotation data to a set of data to be detected.

[0060] The data annotation task is an important step in the field of machine learning and artificial intelligence, especially when training supervised learning models. The task of data annotation is to add meaningful tags or labels to the original data so that the machine learning model can learn patterns from these tags and make predictions or decisions. The data annotation task can include one of an image annotation task, a text annotation task, a voice annotation task, a video annotation task, etc.

[0061] Specifically, when the data labeling task is an image labeling task, the data labeling task may be a task of labeling the categories to which objects in the image belong, such as a task of labeling the breed of a cat in the image; when the data labeling task is a text labeling task, the data labeling task may be a task of labeling the emotion categories to which text belongs, such as labeling the emotion categories to which the text "The weather is really nice today" belongs; when the data labeling task is a speech labeling task, the data labeling task may be a task of labeling the emotion categories to which a speech belongs, or a task of labeling entities in a speech, such as a task of labeling the names of people, places and behaviors in the speech "Xiao Ming went to the museum today and visited dinosaur fossils"; when the data labeling task is a video labeling task, the data labeling task may be a task of identifying the actions of objects in the video, such as labeling the number of shots made by each player in a video of a basketball game.

[0062] In an embodiment of the present application, the above-mentioned data labeling task can be first delivered to a preset number of multiple first labeling clients, and the preset number can be 2, 3, 4, etc., and the present application does not impose any restrictions on this, so that the labeler holding the first labeling client can label the data labeling task. In this way, the first labeling data of each first labeling client for the data labeling task can be obtained, and the labeling results of the first labeling client for the data labeling task can be obtained.

[0063] As shown in Table 1, it is an example table of an example of first annotation data for a data annotation task by a first annotation client in the data annotation method provided in an embodiment of the present application.

[0064] Table 1.

[0065]

[0066] In the above Table 1, the data annotation task is an image annotation task, and a picture of a cat is provided. The question is "What breed is the cat in the picture?" and two options are provided: "A: Ragdoll" and "B: Silver Gradient". The data annotation task is delivered to three first annotation clients, namely, Client 1, Client 2, and Client 3. Among them, Client 1 selects option A for the data annotation task, and believes that the cat in the picture is a Ragdoll cat, that is, the first annotation data corresponding to Client 1 is A; Client 2 selects option A for the data annotation task, and also believes that the cat in the picture is a Ragdoll cat, that is, the first annotation data corresponding to Client 2 is A; Client 3 selects option B for the data annotation task, and believes that the cat in the picture is a Silver Gradient, that is, the first annotation data corresponding to Client 3 is B.

[0067] After the first labeled data is obtained, the first labeled data may be added to the set of data to be detected.

[0068] As shown in Table 2, it is an example table of an example of a data set to be detected in the data labeling method provided in an embodiment of the present application.

[0069] Table 2.

[0070]

[0071]

[0072] Table 2 is a set of data to be detected obtained on the basis of the above Table 1. Based on the first labeled data in Table 1, the set of data to be detected is (A, A, B).

[0073] Step S102: Determine whether the data in the to-be-detected data set meets a preset condition;

[0074] In an embodiment of the present application, the above-mentioned data labeling task may include at least a first delivery stage, and the first labeling client is a client delivered in the first delivery stage. Accordingly, the first labeling data is the labeling result corresponding to the first delivery stage.

[0075] After obtaining the data set to be detected, it can be determined whether the data in the data set to be detected meets a preset condition. The preset condition is a convergence condition of the data labeling task, for example, the preset condition may be that the data in the data set to be detected reaches a preset confidence.

[0076] When the data set to be detected that only contains the first labeled data meets the preset conditions, the data labeling task does not need to go into the second delivery phase, and the labeling result of the data labeling task can be determined directly based on the data in the data set to be detected (ie, the first labeled data).

[0077] When the data in the to-be-detected data set that only includes the first labeled data does not meet the preset condition, the data labeling task needs to proceed to the second delivery stage, which is specifically the following steps S103 to S105:

[0078] Step S103: When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0079] The second labeling client refers to the client delivered in the second delivery phase of the data labeling task. When the data in the to-be-detected data set containing only the first labeling data does not meet the preset conditions, the second delivery phase can be continued to deliver the data labeling task to the second labeling client.

[0080] In the embodiment of the present application, when the data in the to-be-detected data set does not meet the preset conditions, a first annotation request for the data annotation task may be sent to the second annotation client, and the first annotation request may include a request for annotating the data annotation task and a request for parsing the annotated result. In this way, after the second annotation client receives the data annotation task, it may annotate the data annotation task and parse the annotated result.

[0081] It should be noted that the first delivery stage of the data annotation task can be understood as the stage of delivering the data annotation task that is not configured as the parsing mode to the above-mentioned multiple first annotation clients, and the above-mentioned step S101 can be understood as: obtaining the first annotation data of the multiple first annotation clients for the data annotation task that is not configured as the parsing mode. The above-mentioned step S103 can be understood as: when the data in the data set to be detected does not meet the preset conditions, delivering the data annotation task configured as the parsing mode to the second annotation client, so that the annotator can annotate the data annotation task through the second annotation client while parsing the annotated results.

[0082] Step S104: obtaining second annotation data for the data annotation task and parsed content for the second annotation data from the second annotation client, so as to display at least part of the parsed content in a second annotation client that is subsequently launched;

[0083] After each second annotation client completes the annotation, the second annotation data for the data annotation task and the analysis content for the second annotation data can be obtained. The second annotation data represents the annotation result of the second annotation client for the data annotation task, and the analysis content represents the reason for annotating the data annotation task as the second annotation data.

[0084] As shown in Table 3, it is an example table of the second annotation data of the second annotation client for the data annotation task and the parsing content of the second annotation data in the data annotation method provided in the embodiment of the present application.

[0085] Table 3.

[0086]

[0087] In the above Table 3, the second annotation client is Client 4. Client 4 selects option A for the data annotation task and believes that the cat in the picture is a ragdoll cat. That is, the second annotation data corresponding to Client 4 is A. The analysis content of Client 4 is: the cat in the picture has dark blue eyes and is a ragdoll cat.

[0088] It should be noted that in an embodiment of the present application, when the analysis content of the second annotation data is obtained by the second annotation client, in the subsequent delivery process, at least part of the obtained analysis content can be displayed to the second annotation client that is subsequently delivered, so that subsequent annotators can refer to the displayed analysis content when annotating through the second annotation client, thereby assisting the subsequent second annotation client to make more accurate judgments and improve the accuracy of the annotation results. In this way, gradually consistent and highly accurate annotation data can be quickly obtained, thereby achieving rapid convergence of the data annotation task.

[0089] Step S105: adding the second annotated data to the set of data to be detected, and returning to the step of determining whether the set of data to be detected meets a preset condition, until the set of data to be detected meets the preset condition;

[0090] In the embodiment of the present application, when each second annotation client completes the annotation and obtains the second annotation data for the data annotation task, the second annotation data can be added to the above-mentioned data set to be detected, thereby updating the data set to be detected.

[0091] As shown in Table 4, this is an example table of an example of updating a data set to be detected in the data labeling method provided in an embodiment of the present application.

[0092] Table 4.

[0093] Data set to be tested Second annotation data Updated data set to be tested (A, A, B) A (A, A, B, A)

[0094] The above Table 4 updates the data set to be detected based on the data set to be detected as illustrated in the above Table 2 and the second annotation data as illustrated in the above Table 3, wherein the data set to be detected is (A, A, B). When the second annotation data "A" for the data annotation task is obtained, the second annotation data is added to the data set to be detected, thereby obtaining an updated data set to be detected (A, A, B, A).

[0095] In the embodiment of the present application, when the acquired second labeled data is added to the data set to be detected to obtain a new data set to be detected, the above step S102 can be returned to be executed until the data in the obtained data set to be detected meets the above preset conditions.

[0096] Step S106: Determine the labeling result of the data labeling task according to the data in the to-be-detected data set.

[0097] In the embodiment of the present application, when the data set to be detected meets the above preset conditions, the labeling result of the above data labeling task can be determined according to the data in the data set to be detected. Specifically, the following implementation methods may be included:

[0098] In an optional implementation, the above step S106 refers to: determining the labeled data with the highest frequency of occurrence in the data set to be detected as the labeling result of the data labeling task.

[0099] The implementation method is introduced in combination with Table 2 and Table 4. Assuming that the data set to be detected in Table 2 (A, A, B) meets the above preset conditions, option A with the highest frequency of occurrence can be used as the annotation result of the data annotation task "What breed is the cat in the picture?" Assuming that the data set to be detected in Table 4 (A, A, B, A) meets the above preset conditions, option A with the highest frequency of occurrence can be used as the annotation result of the data annotation task "What breed is the cat in the picture?"

[0100] In another optional implementation, the above step S106 refers to: obtaining the weight of the annotation client corresponding to each annotated data in the data set to be detected; and determining the annotation result of the data annotation task according to the each annotated data and the corresponding weight.

[0101] In this implementation, when each annotated data is added to the data set to be detected, the weight of the annotation client corresponding to each annotated data can be recorded. In this way, when the data in the set to be detected meets the preset conditions, the annotation result of the data annotation task can be determined according to each annotated data in the data set to be detected and the corresponding weight. Specifically, the weighted votes of each annotated data can be determined according to the weight of each annotated data and the number of elements of each annotated data in the data set to be detected, and the annotated data with the highest weighted votes is determined as the annotation result of the above data annotation task.

[0102] Optionally, the first labeled client has a first weight, the second labeled client has a second weight, the first weight is less than the second weight, and the sum of the first weight and the second weight is 1.

[0103] Take Table 4 as an example. Assume that the data set to be detected (A, A, B, A) meets the preset conditions, the first weight is 30%, and the second weight is 70%. Then, there are 2 first annotation clients who choose option A, and 1 second annotation client who chooses option A. The weighted votes are 2*30%+1*70%=1.3. There is 1 first annotation client who chooses option B, and 0 second annotation client who chooses option B. The weighted votes are 1*30%+0*70%=0.3. In this way, option A can be determined as the annotation result of the data annotation task "What breed is the cat in the picture?"

[0104] In this setting method, the first annotation client that annotates the data annotation task without referring to the parsed content is set to a lower weight, and the second annotation client that annotates the data annotation task with reference to the parsed content is set to a higher weight. Since the second annotation client that refers to the parsed content has more annotation auxiliary information when annotating, the corresponding second annotation data has a higher accuracy rate than the first annotation data, which can improve the contribution of the second annotation data to the annotation results of the data annotation task, thereby improving the accuracy and reliability of the annotation results of the data annotation task.

[0105] Optionally, each annotation client may have corresponding user portrait information, and the weight of each annotation client is determined based on the user portrait information and the data annotation task. Among them, the user portrait information with a strong correlation with the data annotation task may have a higher weight.

[0106] The user profile of the annotation client refers to the information that characterizes the typical characteristics and behavior patterns of the annotators, which is created by collecting and analyzing the user data of the annotators with the permission of the annotators. For example, the user profile information of Annotator 1 is: part-time student, 20 years old, female, intermediate computer skills, familiar with basic office software, good at annotating animal species; the user profile information of Annotator 2 is: freelancer, 30 years old, male, advanced computer skills, rich experience in using software, good at annotating sports videos.

[0107] Take Tables 1 to 4 above as examples. Assume that the data set to be detected (A, A, B, A) meets the preset conditions, the weight of client 1 in the first annotation client is 60%, the weight of client 2 is 70%, the weight of client 3 is 40%, and the weight of client 4 in the second annotation client is 50%. Then, the weighted votes for option A are 1*60%+1*70%+1*50%=1.8, and the weighted votes for option B are 1*40%=0.4. In this way, option A can be determined as the annotation result of the data annotation task "What breed is the cat in the picture?"

[0108] This setting method determines the weight of each annotation client through the correlation between the user portrait information and the data annotation task. The client with stronger correlation has a higher weight, and the client with weaker correlation has a lower weight. As a result, the annotation data of the annotation client that is good at the data annotation task will contribute more to the annotation result than the annotation data of the annotation client that is not good at the data annotation task, thereby improving the accuracy and reliability of the annotation results of the data annotation task.

[0109] Optionally, in an embodiment of the present application, historical annotation task information of each annotation client can also be obtained, the historical annotation task information including the types of historical annotation tasks and the number of each type. The weight of each annotation client is determined based on the number of targets of the target type corresponding to the data annotation task in the historical annotation task information, wherein the larger the number of targets, the higher the weight.

[0110] The data annotation method provided in an embodiment of the present application includes the following steps: obtaining first annotation data for a data annotation task respectively from multiple first annotation clients; adding the first annotation data to a data set to be detected, and determining whether the data set to be detected meets a preset condition; when the data set to be detected does not meet the preset condition, sending a first annotation request for the data annotation task to a second annotation client, the first annotation request including a request for annotation of the data annotation task and a request for parsing the annotated result; obtaining second annotation data for the data annotation task from the second annotation client, and parsed content for the second annotation data, so as to display at least part of the parsed content in a second annotation client that is subsequently launched; adding the second annotation data to the data set to be detected, and returning to execute the step of determining whether the data set to be detected meets the preset condition until the data set to be detected meets the preset condition; determining the annotation result of the data annotation task based on the data in the data set to be detected.

[0111] It can be seen that the data annotation method provided in the embodiment of the present application, when the first annotation data of the data annotation task in the first annotation client does not meet the preset conditions, then when the data annotation task is delivered to the second annotation client, the second annotation client is requested to annotate and the annotation result is analyzed and explained. In this way, when the second annotation data for the data annotation task and the analysis content of the second annotation data are obtained from the second annotation client, the analysis content is presented to the subsequent second annotation client, thereby assisting the subsequent second annotation client to use the analysis content as a reference for more accurate annotation. In this way, annotation data that gradually tends to be consistent and has a higher degree of accuracy can be quickly obtained, thereby achieving rapid convergence of the data annotation task. Therefore, the data annotation method provided in the embodiment of the present application can reduce the cost of data annotation and improve the efficiency and accuracy of the annotation results.

[0112] In an optional implementation, the “determining whether the to-be-detected data set meets a preset condition” in the above step S102 may include the following steps S1021 and S1022:

[0113] Step S1021: Determine whether the data set to be detected meets a preset confidence level;

[0114] Step S1022: If the data set to be detected meets the preset confidence level, determine that the data set to be detected meets the preset condition.

[0115] In the embodiment of the present application, a confidence level to be achieved can be preset for the data annotation task, and the preset confidence level to be achieved is also a preset confidence level, for example, the preset confidence level is 95%. In this way, when the data set to be detected is obtained, it can be determined whether the data set to be detected meets the preset confidence level. When the data set to be detected meets the preset confidence level, it is determined that the data set to be detected meets the preset condition.

[0116] Optionally, the above step S1021 can be implemented by the following steps S201 to S203:

[0117] Step S201: determining a confidence value corresponding to the preset confidence level;

[0118] Step S202: determining the confidence interval of the to-be-detected data set according to the confidence value;

[0119] Step S203: If the width of the confidence interval is less than or equal to a preset width, it is determined that the data set to be detected meets the preset confidence level.

[0120] In statistics, the confidence value corresponding to the confidence level refers to the critical value used when calculating the confidence interval. This value is the key value used to determine the range of the confidence interval. Different confidence levels correspond to different confidence values. The confidence level is usually expressed as a percentage, such as 90%, 95%, 99%, etc. The confidence value is a specific value found in the standard normal distribution (Z distribution) or t distribution based on the selected confidence level. This value is used to determine the width of the confidence interval.

[0121] In practical applications, the confidence value corresponding to the preset confidence level can be obtained based on the standard normal distribution table. The confidence value corresponding to the 90% confidence level is approximately 1.645; the confidence value corresponding to the 95% confidence level is approximately 1.96; and the confidence value corresponding to the 99% confidence level is approximately 2.576.

[0122] After obtaining the confidence value corresponding to the preset confidence level, the confidence interval of the data set to be tested can be determined based on the confidence value. The confidence interval is an interval estimation method used in statistics to estimate the population parameter. The confidence interval provides a range that contains the true value of the unknown population parameter with a certain probability (i.e., the confidence level). The two endpoints of the confidence interval are called confidence limits, which are the upper and lower limits of the confidence interval.

[0123] In the embodiment of the present application, the width of the confidence interval can be preset, and the width of the preset confidence interval is the preset width. If the width of the confidence interval calculated in step S202 is less than or equal to the preset width, it is determined that the corresponding data set to be detected meets the preset confidence level. For example, if the calculated width of the confidence interval of the data set to be detected is 0.5 and the preset width is 0.6, it means that the data set to be detected meets the preset confidence level.

[0124] If the width of the confidence interval calculated in step S202 is greater than the preset width, it is determined that the corresponding data set to be detected does not meet the preset confidence level. For example, if the width of the confidence interval of the data set to be detected is 1 and the preset width is 0.6, it means that the data set to be detected does not meet the preset confidence level.

[0125] In an embodiment of the present application, the confidence interval of the data set to be detected can be determined by an interval estimation algorithm. Interval estimation is a statistical method for estimating the possible range of unknown parameters, which is used to measure the reliability of the estimate given a preset confidence level. The interval estimation algorithm specifically estimates an interval of the overall parameter based on sample data, so that this interval contains the true parameter value with a certain probability (i.e., a given preset confidence level).

[0126] For different types of data labeling tasks, the above interval estimation algorithm can be applied through different specific implementations to determine the corresponding confidence interval.

[0127] In an optional specific implementation mode 1, the data labeling task is a task of inputting a numerical value, and the step S202 can be implemented in the following manner:

[0128] Determine the mean and standard deviation of the data set to be detected;

[0129] A confidence interval of the to-be-detected data set is determined according to the confidence value, the mean value, and the standard deviation.

[0130] When the data labeling task is a task of inputting numerical values, the mean and standard deviation of the data set to be tested can be determined according to the numerical values ​​of each labeled data in the data set to be tested. Then, based on the confidence value, mean, standard deviation and the number of elements in the data set to be tested, the confidence interval of the data set to be tested can be determined.

[0131] In the embodiment of the present application, when the data annotation task is a task of inputting numerical values, the confidence interval of the data set to be detected can be calculated by the following formula (1):

[0132]

[0133] In the above formula (1), C1 is the confidence interval of the data set to be tested when the data labeling task is a task of inputting numerical values. is the mean of the data set to be detected, Z is the confidence value corresponding to the preset confidence level, σ is the standard deviation of the data set to be detected, and n is the number of elements in the data set to be detected.

[0134] For example, the data set to be tested includes 100 labeled data, the mean of the 100 labeled data is 50, and the standard deviation is 10. Then the confidence interval corresponding to the 95% confidence level can be calculated by the above formula (1) as [48.04, 51.96].

[0135] In an optional specific implementation mode 2, the data labeling task is a task of selecting an option, and the step S202 can be implemented in the following manner:

[0136] Determine the probability of each option in the data set to be detected;

[0137] The confidence interval of the data set to be detected is determined according to the confidence value and the probability of each option.

[0138] When the data labeling task is a task of selecting options, the probability of each option in the data set to be tested can be counted first, and then the confidence interval of the data set to be tested can be determined based on the confidence value, the probability of each option and the number of elements in the data set to be tested.

[0139] In the embodiment of the present application, when the data annotation task is a task of selecting an option, the confidence interval of a certain option in the data set to be detected can be calculated by the following formula (2):

[0140]

[0141] In the above formula (2), C2 is the confidence interval of the data set to be tested when the data labeling task is a task of selecting options, p is the probability of a certain option in the data set to be tested, Z is the confidence value corresponding to the preset confidence level, and n is the number of elements in the data set to be tested.

[0142] For example, the data set to be tested includes 100 annotated data, of which 50 are option A and 50 are option B. Then, the selection ratio of option A can be calculated by the above formula (2) with a 95% confidence interval of [0.402, 0.598], and the selection ratio of option B can also be calculated with a 95% confidence interval of [0.402, 0.598].

[0143] Through the above method, it is determined whether the data set to be detected meets the preset confidence level. When the confidence level is met, it means that the data set to be detected has converged and a unique and definite labeling result is obtained.

[0144] In an optional implementation, the above data labeling task may be set with a maximum delivery quantity, and the data labeling method provided in the embodiment of the present application may further include the following steps:

[0145] If the to-be-detected data set does not satisfy the preset confidence level, obtaining a first number of annotated clients that have completed the annotating;

[0146] If the first quantity reaches the maximum delivery quantity, it is determined that the data to be detected meets the preset condition.

[0147] In the embodiment of the present application, the delivery of data annotation tasks can be constrained by presetting the confidence level or the maximum delivery quantity. The maximum delivery quantity can be, for example, 50, 80, 100, etc., which can be specifically set based on actual conditions, and the present application does not limit this.

[0148] If the width of the confidence interval calculated in step S202 is greater than the preset width, it is determined that the data set to be detected does not meet the above preset confidence level. At this time, the first number of annotated clients that have completed the annotation can be obtained. For example, the number of annotated clients that have completed the annotation is currently 10. If the first number reaches the maximum number of delivery, it is determined that the data set to be detected meets the preset condition. At this time, the delivery of the data annotation task is stopped, and the annotation result of the data annotation task is determined based on the data in the data set to be detected.

[0149] In the embodiment of the present application, it is first determined whether the set to be detected meets the preset confidence level. If the preset confidence level is met, a labeling result with high accuracy and consistency is obtained, and the delivery of the data labeling task can be stopped. When the preset confidence level is not met, it is determined whether the number of labeling clients that have been delivered has reached the maximum delivery number set for the data labeling task. If so, the delivery of the data labeling task is stopped, which can solve the problem of high cost caused by the delivery of a large number of data labeling tasks.

[0150] In an optional implementation, the above step S103 can be implemented by the following steps:

[0151] Determining, based on the to-be-detected data set, that when the preset confidence level is reached, at least a second number of the data labeling tasks need to be deployed;

[0152] If the sum of the second number and the first number is greater than the maximum delivery number, a first labeling request for the data labeling task is sent to a second labeling client.

[0153] In the embodiment of the present application, it can be determined based on each annotated data in the data set to be detected that when the preset confidence level is reached, at least the second number of data annotation tasks need to be launched. That is, based on the sample data of the current data set to be detected, at least the second number of data annotation tasks need to be launched, and after the obtained annotated data is added to the data set to be detected, the new data set to be detected can reach the preset confidence level.

[0154] Optionally, the step of “determining, based on the data set to be detected, that at least a second number of data labeling tasks need to be placed when the preset confidence level is reached” can be implemented by the following steps:

[0155] Determine the target annotation data with the highest frequency in the data set to be detected;

[0156] Determine in a first state, when the preset confidence level is reached, the second quantity required to be put into the data labeling task, the first state being a state in which a subsequent labeling result is the target labeling data.

[0157] In the embodiment of the present application, the target annotation data with the highest frequency in the data set to be detected can be determined first. In the first state, the annotation data corresponding to the second annotation client subsequently delivered are all the target annotation data, so that the subsequent annotation data can meet the above preset confidence level as quickly as possible after being added to the data set to be detected.

[0158] Taking Table 1 as an example, in the data set to be detected in Table 1 (A, A, B), the target annotation data with the highest frequency is A. When the annotation data of the subsequent second annotation client are all A, the preset confidence level can be met by delivering it to 50 people, and the second number is 50.

[0159] After obtaining the second number, if the sum of the second number and the first number of annotation clients that have currently completed annotation is greater than the above-mentioned maximum delivery number, a first annotation request for the data annotation task is sent to the second annotation client, that is, a data annotation task configured in parsing mode is sent to the second annotation client, so that the second annotation client can input parsing content for the annotated data while annotating the data annotation task, so as to present it to subsequent annotators as auxiliary annotation information, thereby forming a virtuous circle. In this way, rapid convergence can be achieved during the delivery of data annotation tasks, thereby improving the accuracy and efficiency of data annotation.

[0160] As shown in Table 5, this is an example table of an example of a data annotation task placed by the second annotation client in the data annotation method provided in an embodiment of the present application.

[0161] Table 5.

[0162]

[0163] In the above Table 5, the data set to be detected is (A, A, B), that is, the first number of the annotation clients that have completed the annotation is 3. The data set to be detected does not meet the preset confidence level and has not reached the maximum delivery quantity of 50. At this time, when the annotation data of the subsequent annotation clients are all A, it is calculated that at least 50 times of the data annotation task needs to be delivered to different annotation clients to meet the preset confidence level. In this way, the second number is 50, and the sum of the first number and the second number is 53, which is greater than the maximum delivery data. In this way, it is necessary to turn on the parsing mode, and at the same time as delivering the data annotation task to the subsequent second annotation client, request the subsequent second annotation client to input the parsing, so that the subsequent second annotation client can input the reason for selecting the corresponding option while annotating the data annotation task.

[0164] With this setting, when the sum of the currently calculated at least required second quantity and the first quantity of completed annotations is greater than the maximum number of deliveries, it means that a large number of deliveries of the data annotation task are required to meet the preset confidence level and obtain an accurate and unique annotation result. In this way, after sending a first annotation request including annotation and analysis for the data annotation task to the second annotation client, the analysis content can be collected and presented to the subsequent second annotation client, thereby assisting the subsequent second annotation client to more accurately annotate the data annotation task based on the analysis content of the previous annotator, thereby quickly making the annotation data consistent and avoiding the problem of excessive cost caused by excessive number of deliveries required to meet the preset confidence level.

[0165] In an optional implementation, the data labeling method provided in the embodiment of the present application may further include the following steps:

[0166] If the sum of the second number and the first number is less than or equal to the maximum delivery number, a second labeling request for the data labeling task is sent to a second labeling client, where the second labeling request includes a request for labeling the data labeling task.

[0167] After obtaining the second number, if the sum of the second number and the first number of annotation clients that have currently completed annotation is less than or equal to the above-mentioned maximum delivery number, a second annotation request for the data annotation task is sent to the second annotation client, and the second annotation request only includes a request to annotate the data annotation task, that is, a data annotation task that is not configured in parsing mode is sent to the second annotation client, so that the second annotation client annotates the data annotation task.

[0168] As shown in Table 6, this is an example table of an example of a data annotation task placed by the second annotation client in the data annotation method provided in an embodiment of the present application.

[0169] Table 5.

[0170]

[0171] In the above Table 5, the data set to be detected is (A, A, B), that is, the first number of annotation clients that have completed the annotation is 3. The data set to be detected does not meet the preset confidence level and has not reached the maximum number of deliveries of 50. At this time, when the annotation data of the subsequent annotation clients are all A, it is calculated that at least 40 times of delivery of the data annotation task to different annotation clients are required to meet the preset confidence level. In this way, the second number is 40, and the sum of the first number and the second number is 43, which is less than the maximum delivery data. In this way, there is no need to turn on the parsing mode, and it is only necessary to deliver the data annotation task that does not contain the parsing request to the subsequent second annotation client, so that the subsequent second annotation client only needs to label the data annotation task without entering the parsing content.

[0172] With this setting, when the sum of at least the required second number currently calculated and the first number of completed annotations is less than or equal to the maximum number of deliveries, it means that within a smaller number of deliveries, the preset reliability can be met and an accurate and unique annotation result can be obtained. In this way, there is no need to turn on the parsing mode. The second annotation client only needs to send a second annotation request including annotations for the data annotation task, so that convergence can be quickly achieved within a smaller number of deliveries.

[0173] It can be seen that the data labeling method provided in the embodiment of the present application flexibly controls whether to configure the data labeling task as the parsing mode based on the current labeling situation, thereby quickly obtaining accurate and consistent labeled data within a smaller number of deliveries, greatly reducing the cost of data labeling, and at the same time improving the efficiency and accuracy of data labeling.

[0174] In an optional implementation, the “sending a first annotation request for the data annotation task to the second annotation client” in the above step S103 can be implemented by the following steps:

[0175] For the first delivered second labeling client, sending a first labeling request for the data labeling task to the first delivered second labeling client, so as to display the data labeling task through the first delivered second labeling client;

[0176] For the i-th delivered second annotation client, send a first annotation request for the data annotation task and at least a part of the target parsed content to the i-th delivered second annotation client, so as to display the data annotation task and at least a part of the target parsed content through the i-th delivered second annotation client;

[0177] Among them, i traverses 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the i-1th second annotation client.

[0178] In an embodiment of the present application, when the parsing mode is turned on, in the process of delivering a data labeling task configured for the parsing mode to the second labeling client, a first labeling request for the data labeling task can be sent to the first delivered second labeling client, and a labeling area for the data labeling task can be displayed in the graphical user interface of the first delivered second labeling client, and a parsing input area can be provided at the same time, so that the labeler can label the data labeling area in the labeling area through the first delivered second labeling client, and enter the parsing content in the parsing input area.

[0179] For any i-th second annotation client from the second to the N-th second annotation client, the first annotation request for the data annotation task and at least part of the parsing content corresponding to the first to the i-1-th second annotation client can be sent to the i-th second annotation client. In this way, in the graphical user interface of the i-th second annotation client, the annotation area of ​​the data annotation task can be displayed, and a parsing input area can be provided at the same time, and at least part of the target parsing content can be displayed in the annotation area, and specifically, the at least part of the target parsing content can be presented in the form of bullet screen.

[0180] In an optional implementation, a display strategy for displaying the analysis contents corresponding to different annotated data can be used to display at least part of the target analysis contents in the i-th delivered second annotation client. For example, if the data annotation task is an annotation task for selecting options, including option A and option B, the analysis contents corresponding to option A can be displayed in red in the i-th delivered second annotation client, and the analysis contents corresponding to option B can be displayed in blue.

[0181] It should be noted that the present application is not limited to the display strategy of distinguishing the analysis content corresponding to different annotation data with different colors. It can also be a display strategy of distinguishing the analysis content corresponding to different annotation data with different fonts. It can also be a display strategy of distinguishing the analysis content corresponding to different annotation data with different display positions. The present application does not limit this.

[0182] In this way, the parsed content input by the previous client can be intuitively displayed to the subsequent second annotation client, and the display strategy of the parsed content corresponding to different annotation data can be distinguished, which can provide visual guidance for subsequent annotators, so that the annotators can obtain important information in the parsed content corresponding to different annotation data in a timely manner. In addition, the annotators can clearly understand the selection situation corresponding to each annotation data, which facilitates users to make quick decisions and further improves the efficiency of data annotation.

[0183] The following, combined Figure 3 The whole process of the data annotation method provided in the first embodiment of the present application is introduced, including the following steps S301 to S311:

[0184] Step S301: sending a second annotation request for a data annotation task to multiple first annotation clients;

[0185] Step S302 (i.e. Figure 2 Step S101): obtaining first annotation data of a plurality of first annotation clients for the data annotation task, and adding the first annotation data to a set of data to be detected;

[0186] Step S303 (ie, the aforementioned step S1021): determining whether the current data set to be detected meets a preset confidence level;

[0187] If the result of step S303 is no, execute step S304: determine whether the first number of currently completed markings reaches the maximum number of delivery

[0188] If the result of step S304 is no, step S305 is executed: determining that at least a second quantity needs to be delivered when the preset confidence level is met;

[0189] Step S306: Determine whether the sum of the second quantity and the first quantity that has been marked is greater than the maximum delivery quantity;

[0190] If the result of step S306 is yes, execute step S307: send a first annotation request for the data annotation task to the second annotation client;

[0191] Step S308 (i.e., the aforementioned steps S104 and S105): obtaining the second annotation data for the data annotation task and the parsed content for the second annotation data from the second annotation client, so as to present the parsed content to the subsequent second annotation client, and adding the second annotation data to the set of data to be detected; and returning to execute step S303;

[0192] If the result of step S306 is yes, execute step S309: send a second annotation request for the data annotation task to the second annotation client;

[0193] Step S310: obtaining second annotation data for the data annotation task from the second annotation client, and adding the second annotation data to the set of data to be detected; and returning to execute step S303;

[0194] If the result of step S303 is yes, execute step S311: determine the labeling result of the data labeling task according to the data in the data set to be detected;

[0195] If the result of step S304 is yes, the above step S311 is executed.

[0196] The above steps S301 to S311 can refer to the relevant introduction in the above embodiments, which will not be described in detail here.

[0197] In the embodiment of the present application, each time the data set to be detected is updated, it can be determined whether the data set to be detected meets the preset confidence level, and the delivery is stopped when the preset confidence level is met, and the labeling result of the data labeling task is determined according to the data in the data set to be detected. When the preset confidence level is not met, it is determined whether the first number of the currently labeled clients that have completed the labeling reaches the maximum delivery number, and the delivery is stopped when the maximum delivery number is reached, and the labeling result of the data labeling task is determined according to the data in the data set to be detected. When the maximum delivery quantity is not reached, the second quantity of the data annotation task that is at least required to be delivered when the preset confidence level is currently met is calculated. When the sum of the second quantity and the first quantity is greater than the maximum delivery quantity, the parsing mode is turned on, and a first labeling request for the data annotation task is sent to the second labeling client. The first labeling request is a request including labeling for the data annotation task and parsing for the labeled data, so that the second labeling client that performs labeling later inputs the parsing content for the labeled data while labeling the data annotation task, and the parsing content input is displayed to the second labeling client that labels later, so that the second labeling client that labels later can refer to the parsing content for the labeled data of the second labeling client that previously labeled, and make a correct judgment, so as to quickly make the labeled data consistent, and avoid the problem of excessive cost caused by too many delivery quantities required to meet the preset confidence level. When the sum of the second quantity and the first quantity is less than or equal to the maximum delivery quantity, a second labeling request for the data annotation task is sent to the second labeling client, and the second labeling request is a request including labeling for the data annotation task and no need to parse the labeled data, so that the second labeling client that performs labeling later labels the data annotation task.

[0198] The second embodiment of the present application provides a display method, such as Figure 4 FIG. 4 is a flowchart of a display method provided in the second embodiment of the present application, including steps S401 to S403:

[0199] Step S401: in response to receiving a first annotation request for a data annotation task, displaying an annotation area and a parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0200] Step S402: in response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task;

[0201] Step S403: in response to the input operation in the analysis input area, obtaining analysis content for the second annotation data input, so as to display at least part of the analysis content in a second annotation client that is subsequently delivered.

[0202] The above-mentioned display method can be applied to a second annotation client. When the second annotation client receives a first annotation request including annotation for a data annotation task and analysis of the annotated results, the annotation interface of the data annotation task can be displayed through the graphical user interface of the second annotation client. The annotation interface includes an annotation area for annotating the data annotation task and a analysis input area for analyzing the annotated results.

[0203] The annotator holding the second annotation client can perform an annotation operation in the annotation area, and upon receiving the annotation operation, the second annotation data for the data annotation task can be obtained. In addition, the annotator holding the second annotation client can perform an input operation in the parsing input area, and upon receiving the input operation, the parsing content input for the second annotation data can be obtained, so as to display at least part of the parsing content in the second annotation client that is subsequently launched.

[0204] Optionally, the analysis input area may include at least one of a preset analysis option area and a custom analysis content area. The preset analysis content area includes a plurality of optional preset analysis options, from which the user can select a target analysis option, which is the input analysis content. The custom analysis content area is used to support the user to input custom analysis content.

[0205] The preset analysis options may be a preset number of analysis contents that appear frequently in the currently acquired analysis contents, for example, the preset analysis options are the first 10 analysis contents that appear frequently in the currently acquired analysis contents.

[0206] The display method provided by the second embodiment of the present application includes the following steps: in response to receiving a first annotation request for a data annotation task, displaying the annotation area and the analysis input area of ​​the data annotation task through the second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and analyzing the annotated result; in response to the annotation operation in the annotation area, obtaining the second annotation data for the data annotation task; and in response to the input operation in the analysis input area, obtaining the analysis content input for the second annotation data, so as to display at least part of the analysis content in the second annotation client that is subsequently launched. It can be seen that the display method provided by the embodiment of the present application, when receiving the first annotation request for the data annotation task, can annotate the data annotation task through the second annotation client and input the analysis corresponding to the annotated result by displaying the annotation area and the analysis input area of ​​the data annotation task on the second annotation client, and display at least part of the analysis content in the second annotation client that is subsequently launched. In this way, by displaying the analysis content in the second annotation client for subsequent annotation, it is possible to assist the second annotation client for subsequent annotation to perform more accurate data annotation, thereby improving the efficiency of data annotation.

[0207] In an optional implementation manner, the display method provided in the embodiment of the present application may further include the following steps:

[0208] In response to receiving target parsed content for the data labeling task from other labeling clients, at least a portion of the target parsed content is displayed through the second labeling client.

[0209] Optionally, the annotation area includes a title display area, and the above step of "displaying at least part of the target parsed content through the second annotation client" can be implemented by the following steps:

[0210] At least a portion of the target analysis content is displayed in the question display area in the form of a bullet screen.

[0211] Optionally, the above step of "displaying at least part of the target parsed content through the second annotation client" may be implemented by the following steps:

[0212] The second annotation client is used to display at least a portion of the target parsed content by using a display strategy that distinguishes and displays parsed content corresponding to different annotation data.

[0213] In the second embodiment of the present application, when the second annotation client receives target parsing content for a data annotation task from other annotation clients, at least part of the target parsing content may be displayed in the second annotation client.

[0214] In a specific implementation, the target analysis content includes multiple different first contents. For one first content, the third quantity of the first content in the target analysis content can be obtained, and for the first content, the first content and the corresponding third quantity can be displayed in the above-mentioned question display area in the form of barrage.

[0215] In this embodiment, when the second annotation client receives the target parsed content of the data annotation task from other annotation clients, it can display at least part of the target parsed content in the second annotation client by using a display strategy that distinguishes the parsed content corresponding to different annotation data. This part is introduced in detail in the above introduction to the first embodiment and will not be repeated here.

[0216] like Figure 5 As shown, it is a schematic diagram of an interface of an example of the second annotation client in the display method provided by the second embodiment of the present application. In response to receiving the first annotation request for the data annotation task, the second annotation client displays the annotation area 50 and the analysis input area 51, and the annotation area 50 includes the question display area 52. In the question display area 52, the analysis content previously input by the second annotation client is displayed in the form of bullet screen, including 5 bullet screens for the analysis content of option A "Ragdoll" "Ragdoll cats are large, long, and muscular", 3 bullet screens for the analysis content of option B "Silver Gradient" "Unique red nose, must be Silver Gradient", and 8 bullet screens for the analysis content of option A "Ragdoll" "Dark blue eyes". The analysis input area 51 may include a preset analysis content area and a custom analysis content area. The preset analysis content area includes multiple preset analysis contents, namely "Ragdoll cats are large in size", "Ragdoll cats have broad chests", "well-developed muscles", "short and thick necks", and "dark blue eyes". The annotator can select option A in the annotation area 51 to annotate the data annotation task, and select the preset analysis contents such as "well-developed muscles" and "dark blue eyes" in the preset analysis content area in the analysis input area 51 to input the analysis content for the selected option A.

[0217] Corresponding to the data labeling method provided in the first embodiment of the present application, the third embodiment of the present application further provides a data labeling device, such as Figure 6 As shown, the data labeling device 600 includes:

[0218] A first acquisition unit 601 is used to acquire first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected;

[0219] A first determining unit 602 is used to determine whether the to-be-detected data set meets a preset condition;

[0220] A sending unit 603 is used to send a first annotation request for the data annotation task to a second annotation client when the to-be-detected data set does not meet a preset condition, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0221] A second obtaining unit 604 is used to obtain the second annotation data for the data annotation task and the parsed content for the second annotation data from the second annotation client, so as to display at least part of the parsed content in the second annotation client that is subsequently launched;

[0222] An adding unit 605 is used to add the second labeled data to the set of data to be detected, and return to the step of determining whether the set of data to be detected meets the preset condition until the set of data to be detected meets the preset condition;

[0223] The second determining unit 606 is used to determine the labeling result of the data labeling task according to the data in the to-be-detected data set.

[0224] Optionally, the first determining unit 602 is specifically configured to:

[0225] Determining whether the data set to be detected meets a preset confidence level;

[0226] If the data set to be detected meets the preset confidence level, it is determined that the data set to be detected meets the preset condition.

[0227] Optionally, the data labeling task is set with a maximum delivery quantity, and the first determining unit 602 is further specifically configured to:

[0228] If the to-be-detected data set does not satisfy the preset confidence level, obtaining a first number of annotated clients that have completed the annotating;

[0229] If the first quantity reaches the maximum delivery quantity, it is determined that the data set to be detected meets the preset condition.

[0230] Optionally, the sending unit 603 is specifically configured to:

[0231] Determining, based on the to-be-detected data set, that when the preset confidence level is reached, at least a second number of the data labeling tasks need to be deployed;

[0232] If the sum of the second number and the first number is greater than the maximum delivery number, a first labeling request for the data labeling task is sent to a second labeling client.

[0233] Optionally, the sending unit 603 is further specifically configured to:

[0234] If the sum of the second number and the first number is less than or equal to the maximum delivery number, a second labeling request for the data labeling task is sent to a second labeling client, where the second labeling request includes a request for labeling the data labeling task.

[0235] Optionally, the sending unit 603 is specifically configured to:

[0236] For the first delivered second labeling client, sending a first labeling request for the data labeling task to the first delivered second labeling client, so as to display the data labeling task through the first delivered second labeling client;

[0237] For the i-th delivered second annotation client, send a first annotation request for the data annotation task and at least a part of the target parsed content to the i-th delivered second annotation client, so as to display the data annotation task and at least a part of the target parsed content through the i-th delivered second annotation client;

[0238] Among them, i traverses 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the i-1th second annotation client.

[0239] Optionally, the sending unit 603 is specifically configured to:

[0240] Determine the target annotation data with the highest frequency in the data set to be detected;

[0241] Determine in a first state, when the preset confidence level is reached, the second quantity required to be delivered for the data labeling task, the first state being a state in which subsequent labeled data is the target labeled data.

[0242] Optionally, the first determining unit 602 is specifically configured to:

[0243] Determining a confidence value corresponding to the preset confidence level;

[0244] Determining a confidence interval of the to-be-detected data set according to the confidence value;

[0245] If the width of the confidence interval is less than or equal to the preset width, it is determined that the data set to be detected meets the preset confidence level.

[0246] Optionally, the data labeling task is a task of inputting a numerical value, and the first determining unit 602 is specifically configured to:

[0247] Determine the mean and standard deviation of the data set to be detected;

[0248] A confidence interval of the to-be-detected data set is determined according to the confidence value, the mean value, and the standard deviation.

[0249] Optionally, the data labeling task is a task of selecting options, and the first determining unit 602 is specifically configured to:

[0250] Determine the probability of each option in the data set to be detected;

[0251] The confidence interval of the data set to be detected is determined according to the confidence value and the probability of each option.

[0252] Optionally, the second determining unit 606 is specifically configured to:

[0253] Obtaining the weight of the annotation client corresponding to each annotated data in the to-be-detected data set;

[0254] A labeling result of the data labeling task is determined according to the labeling data and the corresponding weights.

[0255] Corresponding to the display method provided in the second embodiment of the present application, the fourth embodiment of the present application further provides a display device, such as Figure 7 As shown, the display device 700 includes:

[0256] A first display unit 701 is used for displaying a labeling area and a parsing input area of ​​the data labeling task through a second labeling client in response to receiving a first labeling request for the data labeling task, wherein the first labeling request includes a request for labeling the data labeling task and a request for parsing the labeled result;

[0257] The third acquisition unit 702 is used to obtain second annotation data for the data annotation task in response to the annotation operation in the annotation area; and to obtain the analysis content input for the second annotation data in response to the input operation in the analysis input area, so as to display at least part of the analysis content in a second annotation client that is subsequently launched.

[0258] Optionally, the first display unit 701 is further used for:

[0259] In response to receiving target parsed content for the data labeling task from other labeling clients, at least a portion of the target parsed content is displayed through the second labeling client.

[0260] Optionally, the marked area includes a question display area, and the first display unit 701 is further specifically used for:

[0261] At least a portion of the target analysis content is displayed in the question display area in the form of a bullet screen.

[0262] Optionally, the first display unit 701 is specifically used for:

[0263] The second annotation client is used to display at least a portion of the target parsed content by using a display strategy that distinguishes and displays parsed content corresponding to different annotation data.

[0264] Corresponding to the data annotation method provided in the first embodiment of the present application or the display method provided in the second embodiment of the present application, the fifth embodiment of the present application further provides an electronic device for data annotation or for display.

[0265] like Figure 8 , which is a structural block diagram of an example of an electronic device for data annotation or display provided in an embodiment of the present application.

[0266] In this embodiment, an optional hardware structure of the electronic device 800 can be as follows: Figure 8 As shown, it includes: at least one processor 801, at least one memory 802 and at least one communication bus 805; the memory 802 contains a program 803 and data 804.

[0267] The bus 805 may be a communication device for transmitting data between components inside the electronic device 800, such as an internal bus (e.g., a CPU-memory bus, where the processor is a central processing unit, CPU for short), an external bus (e.g., a universal serial bus port, a peripheral component interconnect express port), etc.

[0268] In addition, the electronic device also includes: at least one network interface 806, at least one peripheral interface 807. The network interface 806 provides wired or wireless communication related to an external network 808 (for example, the Internet, an intranet, a local area network, a mobile communication network, etc.); in some embodiments, the network interface 806 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (NFC) adapters, cellular network chips, etc.

[0269] The peripheral interface 807 is used to connect to the peripheral device, and the peripheral device can be the peripheral device 1 ( Figure 8 809 in), peripheral 2 ( Figure 8 810) and peripheral 3 ( Figure 8 811 in the figure). Peripherals are peripheral devices, which may include but are not limited to cursor control devices (such as mice, touch pads or touch screens), keyboards, displays (such as cathode ray tube displays, liquid crystal displays). Displays or light emitting diode displays, video input devices (such as cameras or communication interfaces coupled to video files), etc.

[0270] The processor 801 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0271] The memory 802 may include a high-speed RAM (full name: Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0272] The processor 801 calls the program and data stored in the memory 802 to perform the following steps:

[0273] Obtain first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected;

[0274] Determine whether the data set to be detected meets a preset condition;

[0275] When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0276] Acquire the second annotation data for the data annotation task and the parsed content for the second annotation data by the second annotation client, so as to display at least part of the parsed content in the second annotation client that is subsequently launched;

[0277] Add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition;

[0278] Determining a labeling result of the data labeling task according to the data in the to-be-detected data set;

[0279] or

[0280] In response to receiving a first annotation request for a data annotation task, displaying an annotation area and a parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and to parse the annotated result;

[0281] In response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task; and

[0282] In response to an input operation in the parsing input area, parsing content inputted for the second annotation data is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently delivered.

[0283] In addition to the data labeling method provided in the first embodiment of the present application or the display method provided in the second embodiment of the present application, the sixth embodiment of the present application provides a computer-readable storage medium storing a program of the data labeling method provided in the first embodiment of the present application or the display method provided in the second embodiment of the present application, and the program is executed by a processor to perform the following steps:

[0284] Obtain first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected;

[0285] Determine whether the data set to be detected meets a preset condition;

[0286] When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result;

[0287] Acquire the second annotation data for the data annotation task and the parsed content for the second annotation data by the second annotation client, so as to display at least part of the parsed content in the second annotation client that is subsequently launched;

[0288] Add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition;

[0289] Determining a labeling result of the data labeling task according to the data in the to-be-detected data set;

[0290] or

[0291] In response to receiving a first annotation request for a data annotation task, displaying an annotation area and a parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and to parse the annotated result;

[0292] In response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task; and

[0293] In response to an input operation in the parsing input area, parsing content inputted for the second annotation data is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently delivered.

[0294] It should be noted that, for the detailed description of the display method, data labeling device, display device, electronic device and computer-readable storage medium provided in the second, third, fourth, fifth and sixth embodiments of the present application, reference can be made to the relevant description of the first embodiment of the present application and will not be repeated here.

[0295] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0296] In a typical configuration, a node device in a blockchain includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0297] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0298] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), random access memory (RAM) of other attributes, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage media or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0299] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0300] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A data labeling method, characterized in that: The method comprises: Obtain first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected; Determine whether the data set to be detected meets a preset condition; When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result; Acquire the second annotation data for the data annotation task and the parsed content for the second annotation data by the second annotation client, so as to display at least part of the parsed content in the second annotation client that is subsequently launched; Add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition; Determine the labeling result of the data labeling task according to the data in the to-be-detected data set.

2. The method according to claim 1, characterized in that The determining whether the to-be-detected data set satisfies a preset condition comprises: Determining whether the data set to be detected meets a preset confidence level; If the data set to be detected meets the preset confidence level, it is determined that the data set to be detected meets the preset condition.

3. The method according to claim 2, characterized in that The data labeling task is set with a maximum delivery quantity, and the method further includes: If the to-be-detected data set does not satisfy the preset confidence level, obtaining a first number of annotated clients that have completed the annotating; If the first quantity reaches the maximum delivery quantity, it is determined that the data set to be detected meets the preset condition.

4. The method according to claim 3, characterized in that When the to-be-detected data set does not satisfy a preset condition, sending a first annotation request for the data annotation task to a second annotation client includes: Determining, based on the to-be-detected data set, that when the preset confidence level is reached, at least a second number of the data labeling tasks need to be deployed; If the sum of the second number and the first number is greater than the maximum delivery number, a first labeling request for the data labeling task is sent to a second labeling client.

5. The method according to claim 4, characterized in that The method further comprises: If the sum of the second number and the first number is less than or equal to the maximum delivery number, a second labeling request for the data labeling task is sent to a second labeling client, where the second labeling request includes a request for labeling the data labeling task.

6. The method according to claim 1, characterized in that The sending a first annotation request for the data annotation task to the second annotation client includes: For the first delivered second labeling client, sending a first labeling request for the data labeling task to the first delivered second labeling client, so as to display the data labeling task through the first delivered second labeling client; For the i-th delivered second annotation client, send a first annotation request for the data annotation task and at least a part of the target parsed content to the i-th delivered second annotation client, so as to display the data annotation task and at least a part of the target parsed content through the i-th delivered second annotation client; Among them, i traverses 2 to N, N is the total number of second annotation clients, and the target parsing content includes the parsing content corresponding to the first second annotation client to the i-1th second annotation client.

7. The method according to claim 4, characterized in that The step of determining, based on the to-be-detected data set, that when the preset confidence level is reached, at least a second number of the data labeling tasks needs to be placed includes: Determine the target annotation data with the highest frequency in the data set to be detected; Determine in a first state, when the preset confidence level is reached, the second quantity required to be delivered for the data labeling task, the first state being a state in which subsequent labeled data is the target labeled data.

8. The method according to claim 2, characterized in that: The determining whether the to-be-detected data set meets a preset confidence level includes: Determining a confidence value corresponding to the preset confidence level; Determining a confidence interval of the to-be-detected data set according to the confidence value; If the width of the confidence interval is less than or equal to the preset width, it is determined that the data set to be detected meets the preset confidence level.

9. The method according to claim 8, characterized in that The data labeling task is a task of inputting a numerical value, and determining the confidence interval of the to-be-detected data set according to the confidence value includes: Determine the mean and standard deviation of the data set to be detected; A confidence interval of the to-be-detected data set is determined according to the confidence value, the mean value, and the standard deviation.

10. The method according to claim 8, characterized in that The data labeling task is a task of selecting options, and determining the confidence interval of the to-be-detected data set according to the confidence value includes: Determine the probability of each option in the data set to be detected; The confidence interval of the data set to be detected is determined according to the confidence value and the probability of each option.

11. The method according to claim 1, characterized in that: The step of determining the labeling result of the data labeling task according to the data in the to-be-detected data set includes: Obtaining the weight of the annotation client corresponding to each annotated data in the to-be-detected data set; A labeling result of the data labeling task is determined according to the labeling data and the corresponding weights.

12. A display method, characterized in that: The method comprises: In response to receiving a first annotation request for a data annotation task, displaying an annotation area and a parsing input area of ​​the data annotation task through a second annotation client, wherein the first annotation request includes a request to annotate the data annotation task and to parse the annotated result; In response to a labeling operation in the labeling area, obtaining second labeling data for the data labeling task; and In response to an input operation in the parsing input area, parsing content inputted for the second annotation data is obtained, so as to display at least a portion of the parsing content in a second annotation client that is subsequently delivered.

13. The method according to claim 12, characterized in that The method further comprises: In response to receiving target parsed content for the data labeling task from other labeling clients, at least a portion of the target parsed content is displayed through the second labeling client.

14. The method according to claim 13, characterized in that The annotation area includes a question display area, and the displaying at least a portion of the target parsed content through the second annotation client includes: At least a portion of the target analysis content is displayed in the question display area in the form of a bullet screen.

15. The method according to claim 13, characterized in that The displaying at least part of the target parsed content by the second annotation client includes: The second annotation client is used to display at least a portion of the target parsed content by using a display strategy that distinguishes and displays parsed content corresponding to different annotation data.

16. A data labeling device, characterized in that: The device comprises: A first acquisition unit, configured to acquire first annotation data for data annotation tasks respectively from a plurality of first annotation clients, and add the first annotation data to a set of data to be detected; A first determining unit, used to determine whether the to-be-detected data set meets a preset condition; A sending unit, configured to send a first annotation request for the data annotation task to a second annotation client when the to-be-detected data set does not satisfy a preset condition, wherein the first annotation request includes a request for annotating the data annotation task and parsing the annotated result; A second acquisition unit, configured to acquire second annotation data for the data annotation task and parsed content for the second annotation data from the second annotation client, so as to display at least part of the parsed content in a second annotation client that is subsequently launched; an adding unit, configured to add the second annotated data to the set of data to be detected, and return to the step of determining whether the set of data to be detected satisfies a preset condition until the set of data to be detected satisfies the preset condition; The second determining unit is used to determine the labeling result of the data labeling task according to the data in the data set to be detected.

17. A display device, characterized in that: The device comprises: A first display unit, configured to display a labeling area and a parsing input area of ​​the data labeling task through a second labeling client in response to receiving a first labeling request for the data labeling task, wherein the first labeling request includes a request for labeling the data labeling task and for parsing the labeled result; A third acquisition unit is used to obtain second annotation data for the data annotation task in response to an annotation operation in the annotation area; and to obtain parsed content input for the second annotation data in response to an input operation in the parsing input area, so as to display at least part of the parsed content in a second annotation client that is subsequently launched.

18. An electronic device, characterized in that: include: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 11 or claims 12 to 15 is executed.

19. A computer-readable storage medium, characterized in that: A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 11 or claims 12 to 15.

Images