Method, device, computer equipment and storage medium for determining target labeling type
By calculating human-effect data and algorithm derivation functions, we can determine the target annotation type that the annotation object is good at, and solve the problems of low efficiency in calculating human-effect data and difficult to determine the type of annotation personnel in the existing technology, and achieve a more reasonable task allocation and data format consistency.
Patent Information
- Application Number
- CN202410485089.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-22
AI Technical Summary
In the prior art, the calculation of human-effect data is inefficient, and it is difficult to determine the type of labeling personnel good at based on human-effect data, resulting in unreasonable task allocation.
By obtaining the number of labels and the labeling time of the labeling object, the human-effect data calculation formula is used to calculate the human-effect data, and the algorithm derivation function is used to determine the target labeling type that the labeling object is good at, simplifying the calculation process of human-effect data, improving calculation efficiency, and ensuring that Consistency of the result data format.
It improves the efficiency of human-effect data calculation, ensures the accuracy of the results, facilitates the subsequent allocation of labeling tasks more reasonably, and solves the problem of low efficiency in calculating human-effect data and difficulty in determining the type of labeling personnel.
Smart Images

Figure CN118314401B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data annotation technology, and in particular to a method, device, computer equipment and storage medium for determining a target annotation type. Background Art
[0002] Applying artificial intelligence to the vehicle video surveillance system, through artificial intelligence to learn and analyze massive videos and pictures, enables the vehicle video surveillance system to proactively discover vehicle safety risks in real time and issue warning information. Before artificial intelligence learns videos and pictures, the videos and pictures need to be labeled. Currently, data labeling requires labelers, auditors, and reviewers. In this field, the data generated by labelers in the process of labeling data is called human efficiency data, such as: data labeling efficiency, number of data labeling, data labeling time, data labeling data type, etc.
[0003] Currently, the human efficiency data needs to be calculated by the labelers themselves, which consumes the labelers' time and has low calculation efficiency. Human efficiency data can clearly reflect the comprehensive level of the labelers and the types of labeling they are good at. Determining the types of labeling that the labelers are good at is conducive to the subsequent task allocation. However, the current field does not determine the types of labeling that the labelers are good at by analyzing human efficiency data.
[0004] Therefore, the related technology has the problem of low efficiency in calculating human efficiency data and difficulty in determining the type of data that the labelers are good at labeling based on the human efficiency data. Summary of the invention
[0005] In view of this, the present invention provides a method, apparatus, computer device and storage medium for determining a target annotation type to solve the problem of low efficiency in calculating human efficiency data and difficulty in determining the type of data that annotators are good at annotating based on human efficiency data.
[0006] In a first aspect, the present invention provides a method for determining a target annotation type, the method comprising:
[0007] When the first preset number of labeled objects complete the labeling task, obtaining the number of labels and the labeling time of the labeled objects, wherein the data type of the labeling task has a second preset number;
[0008] According to the number of annotations, annotation time and the calculation formula of human efficiency data, the human efficiency data of the annotation object is obtained;
[0009] According to the human efficiency data and the algorithm derivation function, the target annotation type corresponding to the annotation object is determined, wherein the target annotation type is included in a preset number of data types.
[0010] The target labeling type determination method provided in this embodiment calculates the human efficiency data of the labeled object according to the human efficiency data calculation formula, the number of labels, and the labeling time, simplifies the human efficiency data calculation process, improves the efficiency of human efficiency data calculation, and adopts a unified method to calculate the human efficiency data to ensure that the result data format is consistent, which is convenient for subsequent analysis of the human efficiency data. The algorithm derivation function is used to derive the target labeling type that the labeled object is good at based on the human efficiency data, so as to facilitate the subsequent more reasonable allocation of labeling tasks based on the target labeling type. It solves the problem that the related technology has low efficiency in calculating human efficiency data and it is difficult to determine the type of data that the labeler is good at labeling based on the human efficiency data.
[0011] In an optional implementation, obtaining the marking time of the marked object includes:
[0012] Get the start and end time of the annotation task processed by the annotation object;
[0013] When the annotation object is processing the annotation task, if the annotation object suspends the annotation task, obtaining a first moment of suspending the annotation task and a second moment of resuming the annotation task;
[0014] The marked time is obtained according to the start time, the end time, the first time and the second time.
[0015] In this embodiment, the labeling time is calculated according to the start time, end time, first time and second time of the labeling task processed by the labeling object, so that the labeling object does not need to time itself, thereby improving the efficiency of labor efficiency data calculation.
[0016] In an optional implementation, obtaining the number of annotations of the annotation object includes:
[0017] Get the information to be labeled in the labeling task;
[0018] Count the number of information to be annotated and obtain the number of annotations.
[0019] In an optional implementation, the human efficiency data of the labeled object is obtained according to the number of labels, the labeling time, and the human efficiency data calculation formula, including:
[0020] According to the marked object, the number of marked people, the number of audited people and the number of re-examined people are obtained;
[0021] According to the number of annotations, the annotation amount, audit amount, and re-examination amount of each annotation object's completed annotation task, as well as the total annotation amount, audit amount, and re-examination amount of all annotation objects are obtained;
[0022] According to the annotation time, the annotation time, review time, and re-examination time of each annotation object to complete the annotation task, as well as the total annotation time, review time, and re-examination time of all annotation objects are obtained;
[0023] According to the annotation quantity and annotation time, the annotation efficiency of the annotation object is obtained;
[0024] According to the audit volume and audit time, the audit efficiency of the marked object is obtained;
[0025] According to the review quantity and review time, the review efficiency of the marked object is obtained;
[0026] According to the total number of annotations and the number of annotation workers, the average number of annotations per worker is obtained, and according to the total number of annotations and the total annotation time, the average annotation efficiency per worker is obtained.
[0027] Based on the total number of audits and the number of auditors, the average number of audits per capita is obtained, and based on the total number of audits and the total audit time, the average audit efficiency per capita is obtained;
[0028] Based on the total number of reviews and the number of reviewers, the average number of reviews per capita is obtained, and based on the total number of reviews and the total review time, the average review efficiency per capita is obtained;
[0029] The marking efficiency, audit efficiency, re-examination efficiency, average number of markings per capita, average marking efficiency per capita, average number of audits per capita, average audit efficiency per capita, average number of re-examinations per capita, and average re-examination efficiency per capita are summarized to obtain the labor efficiency data.
[0030] In this embodiment, the labor efficiency data is calculated based on the number of annotations, the annotation time, and the labor efficiency data calculation formula, and a standard method is used to calculate the labor efficiency data of different data type annotation tasks. This makes the data format of the annotation platform adopt a unified standard, improves the consistency of the annotation platform, and reduces the complexity of development and maintenance. In addition, the efficiency of labor efficiency data calculation is improved.
[0031] In an optional implementation, determining the target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function includes:
[0032] Based on the labor efficiency data, the labeling efficiency, audit efficiency and review efficiency of each labeling task completed by the labeling object are obtained;
[0033] According to the marking efficiency, the auditing efficiency, the reviewing efficiency and the algorithm derivation function, a first marking task corresponding to the maximum marking efficiency, a second marking task corresponding to the maximum reviewing efficiency and a third marking task corresponding to the maximum reviewing efficiency are obtained;
[0034] The data type of the first labeling task is used as the first target type, the data type of the second labeling task is used as the second target type, and the data type of the third labeling task is used as the third target type, wherein the first target type, the second target type and the third target type are included in the target labeling type.
[0035] In this embodiment, the algorithm derivation function derives the first labeling task corresponding to the maximum labeling efficiency, the second labeling task corresponding to the maximum review efficiency, and the third labeling task corresponding to the maximum review efficiency, and then obtains the first target type, the second target type, and the third target type, which is convenient for determining the data type suitable for labeling, the data type suitable for review, and the data type suitable for review for each labeling object, so as to facilitate more reasonable allocation of labeling tasks in the future.
[0036] In an optional implementation, after determining the target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function, the method further includes:
[0037] Display labor efficiency data and target annotation types on the target page.
[0038] In this embodiment, the labor efficiency data and the target annotation type are displayed on the target page, which is convenient for query and helps to determine the comprehensive level of each annotated object.
[0039] In a second aspect, the present invention provides a device for determining a target annotation type, the device comprising:
[0040] An acquisition module, configured to acquire the number of annotations and the annotation time of the annotation objects when a first preset number of annotation objects complete the annotation task, wherein the data type of the annotation task has a second preset number;
[0041] The obtaining module is used to obtain the human efficiency data of the labeled object according to the number of labels, the labeling time and the human efficiency data calculation formula;
[0042] The determination module is used to determine the target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function, wherein the target annotation type is included in a second preset number of data types.
[0043] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the target labeling type determination method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0044] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the target labeling type determination method of the first aspect or any corresponding embodiment thereof.
[0045] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the target annotation type determination method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 is a schematic diagram of a flow chart of a method for determining a target annotation type according to an embodiment of the present invention;
[0048] Figure 2 is a schematic diagram of a human efficiency data display page according to an embodiment of the present invention;
[0049] Figure 3 is a flowchart of a method for calculating data labeling labor efficiency data according to an embodiment of the present invention;
[0050] Figure 4 is a structural block diagram of a device for determining a target annotation type according to an embodiment of the present invention;
[0051] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0053] The present invention is based on web development, and constructs a labeling page and a labor efficiency statistics page. Taking each labeler as a unit, it displays information such as the number of tasks received within a specified time range and the number of labeling results in the tasks. In addition, according to the degree of granularity of data personnel, the present invention designs three accounts for each labeler who performs data labeling on videos and pictures, namely: a labeling account, an audit account, and a review account. Each labeler will be bound to his or her corresponding account. For example: when a labeler needs to process a labeling task, he or she needs to use the labeling account to log in to the labeling page to process the labeling task; when a review task needs to be processed, the review task is used to log in to the labeling page to process the review task.
[0054] Based on the above content, the embodiment of the present invention provides a method for determining the target annotation type, which first obtains the business data such as the number of annotations and the annotation time of the annotation object; then uses the normalization method to calculate the human efficiency data of the annotation object using the human efficiency data calculation formula; finally, uses the algorithm derivation function to deduce the target annotation type that the annotation object is good at based on the human efficiency data. This simplifies the human efficiency data calculation process and arranges the annotation tasks more reasonably according to the target annotation type.
[0055] According to an embodiment of the present invention, an embodiment of a method for determining a target annotation type is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device with data processing capabilities, such as a computer, a server, etc., and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0056] In this embodiment, a method for determining a target annotation type is provided, which can be used in the above-mentioned computer device. Figure 1 is a flow chart of a method for determining a target annotation type according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0057] Step S101, when a first preset number of labeled objects complete a labeling task, obtaining the labeling number and labeling time of the labeled objects, wherein the data type of the labeling task has a second preset number.
[0058] Specifically, the annotated objects are staff members who are involved in data annotating of videos and images, and the total number of annotated objects is a first preset number, which is determined based on actual conditions. The annotated objects process the annotating tasks through an annotation page in the annotation platform.
[0059] A labeling task is a group of tasks that packages a certain number of pictures and videos of the same data type. A labeling task is a task related to data labeling of videos and pictures, such as labeling pictures and videos, reviewing labeling content, and re-reviewing labeling content. When the labeling object completes the labeling task, the labeling time spent by the labeler and the labeling data completed are obtained. When the labeling object labels pictures and videos, its identity is a labeler. When the labeling object reviews the labeling content, its identity is a reviewer. When the labeling object reviews the labeling content, its identity is a reviewer. The above-mentioned labelers, reviewers, and reviewers are all included in the labeling object. The data type of the labeling task has a second preset number, and the specific value of the second preset number is set according to actual needs. The data types are, for example: face pictures, gesture pictures, parking space pictures, dangerous driving, point clouds, ground signs, etc.
[0060] Step S102, obtaining the human efficiency data of the marked object according to the number of markings, the marking time and the human efficiency data calculation formula.
[0061] Specifically, a series of data generated around data personnel labeling data is human efficiency data, such as: the efficiency of data labeling, the number of data labeling, the time of data labeling, the types of data labeled, etc.
[0062] Substitute the number of annotations and the annotation time into the labor efficiency data calculation formula to calculate the labor efficiency data of each annotation object. For example: the labor efficiency data calculation formula is the formula for calculating the annotation efficiency of the annotation object. Substitute the number of annotated pictures and videos and the consumed annotation time in the number of annotations of the annotation object A into the formula, and divide the number of annotated pictures and videos by the annotation time to get the annotation efficiency of the annotation object A; the labor efficiency data calculation formula is the formula for calculating the average review efficiency of all annotation objects. Substitute the number of reviewed pictures and videos and the consumed annotation time in the number of annotations of all annotators into the formula, first calculate the total number and total time of the pictures and videos reviewed by all annotators, and calculate the average review efficiency by dividing the total number by the total time.
[0063] Step S103: determining a target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function, wherein the target annotation type is included in a preset number of data types.
[0064] Specifically, the target annotation types include: an annotation type that the annotation subject is good at, an audit type that the annotation subject is good at, and a review type that the annotation subject is good at.
[0065] The labeling efficiency, review efficiency and re-review efficiency of the labeling object for labeling tasks of different data types are obtained from the human efficiency data. The algorithm derivation function deduces the labeling type, review type and re-review type that the labeling object is good at according to the labeling efficiency, review efficiency and re-review efficiency. For example: the review efficiency of the labeling object A for the labeling task of reviewing dangerous driving pictures is 500 / hour, the review efficiency for the labeling task of reviewing point cloud pictures is 300 / hour, and the review efficiency for the labeling task of reviewing gesture pictures is 400 / hour. Then the data type that the labeling object A is good at reviewing is dangerous driving. And so on, the same method can be used to deduce the labeling type, review type and re-review type that each labeling object is good at.
[0066] The target labeling type determination method provided in this embodiment calculates the human efficiency data of the labeled object according to the human efficiency data calculation formula, the number of labels, and the labeling time, simplifies the human efficiency data calculation process, improves the efficiency of human efficiency data calculation, and adopts a unified method to calculate the human efficiency data to ensure that the result data format is consistent, which is convenient for subsequent analysis of the human efficiency data. The algorithm derivation function is used to derive the target labeling type that the labeled object is good at based on the human efficiency data, so as to facilitate the subsequent more reasonable allocation of labeling tasks based on the target labeling type. It solves the problem that the related technology has low efficiency in calculating human efficiency data and it is difficult to determine the type of data that the labeler is good at labeling based on the human efficiency data.
[0067] In some optional implementations, obtaining the marking time of the marked object includes:
[0068] Get the start and end time of the annotation task processed by the annotation object;
[0069] When the annotation object is processing the annotation task, if the annotation object suspends the annotation task, obtaining a first moment of suspending the annotation task and a second moment of resuming the annotation task;
[0070] The marked time is obtained according to the start time, the end time, the first time and the second time.
[0071] Specifically, when the annotation object is processing the annotation task, it is necessary to open the annotation page to perform data annotation. This embodiment adds a timing function to the annotation page. When the annotation personnel are annotating data on the annotation page, the program will specifically count their annotation time, and then the database records the specific time consumed by each annotation personnel for the corresponding annotation task.
[0072] When the annotator enters the annotation page to process the annotation task, the annotation page program will obtain the time when the annotated object enters the annotation page as the starting time of the annotation object to process the annotation task. The starting time is, for example, t seconds. When the annotator completes the annotation task on the annotation page, the time at this time is obtained as the end time. For example, when the annotator completes the annotation of all pictures and videos in the annotation task and clicks "Task Completed", the current time is obtained as t 1 +Δt 1 seconds, the end time is t 1 +Δt 1 Second.
[0073] During the process of processing the annotation task, if the annotation object suspends the annotation task, the annotation page program will obtain the time when the annotation object suspends the task as the first time, for example, t+Δt seconds. 1 Resume the annotation task at any time, and record the annotation page program t 1The time is used as the second time to resume the labeling task. The start time, end time, first time and second time are saved.
[0074] If there is a first moment and a second moment, it means that the annotation object has suspended the task when processing the annotation task. Then, the annotation time 1 is calculated based on the start moment and the first moment. For example, if the start moment is t seconds and the first moment is t+Δt seconds, the first moment is subtracted from the start moment to calculate the annotation time 1 as Δt seconds. Then, the annotation time 2 is calculated based on the second moment and the end moment. For example, if the second moment is t 1 , the end time is t 1 +Δt 1 seconds, subtract the second moment from the end moment, and calculate the marked time 2 as Δt 1 The marking time is calculated as Δt by adding the marking time 1 and the marking time 2. 1 +Δt seconds.
[0075] If the first moment does not exist, it means that the annotation object does not suspend the task when processing the annotation task. Then the annotation time is calculated by directly subtracting the start time from the end time. For example, the start time is t seconds and the end time is t seconds. 1 +Δt 1 seconds, then the marked time is t 1 +Δt 1 -t. Save the final calculated annotation time to the database.
[0076] In this embodiment, the labeling time is calculated according to the start time, end time, first time and second time of the labeling task processed by the labeling object, so that the labeling object does not need to time itself, thereby improving the efficiency of labor efficiency data calculation.
[0077] In some optional implementations, obtaining the number of annotations of the annotation object includes:
[0078] Get the information to be labeled in the labeling task;
[0079] Count the number of information to be annotated and obtain the number of annotations.
[0080] Specifically, the annotation object processes the annotation task through the annotation page. This embodiment adds a number counting function to the annotation page. When the annotation object completes the annotation task and saves the processing result, the annotation page program will identify the pictures and videos in the annotation task and save the number of annotations of the pictures and videos to the database.
[0081] When the annotation object completes the annotation task and saves the processing results, the information to be annotated in the annotation task is obtained. For example, after the annotator completes the annotation of all pictures and videos in the annotation task and clicks "Task Completed", the information to be annotated of the annotation task completed by the user is obtained, including: pictures to be annotated, annotated videos, etc. Because the annotation object has completed the annotation task, the number of information to be annotated in all annotation tasks is the number of annotations of the annotation object. Therefore, the number of information to be annotated is counted to obtain the number of annotations.
[0082] In some optional implementations, the human efficiency data of the labeled object is obtained according to the number of labels, the labeling time, and the human efficiency data calculation formula, including:
[0083] According to the marked object, the number of marked people, the number of audited people and the number of re-examined people are obtained;
[0084] According to the number of annotations, the annotation amount, audit amount, and re-examination amount of each annotation object's completed annotation task, as well as the total annotation amount, audit amount, and re-examination amount of all annotation objects are obtained;
[0085] According to the annotation time, the annotation time, review time, and re-examination time of each annotation object to complete the annotation task, as well as the total annotation time, review time, and re-examination time of all annotation objects are obtained;
[0086] According to the annotation quantity and annotation time, the annotation efficiency of the annotation object is obtained;
[0087] According to the audit volume and audit time, the audit efficiency of the marked object is obtained;
[0088] According to the review quantity and review time, the review efficiency of the marked object is obtained;
[0089] According to the total number of annotations and the number of annotation workers, the average number of annotations per worker is obtained, and according to the total number of annotations and the total annotation time, the average annotation efficiency per worker is obtained.
[0090] Based on the total number of audits and the number of auditors, the average number of audits per capita is obtained, and based on the total number of audits and the total audit time, the average audit efficiency per capita is obtained;
[0091] Based on the total number of reviews and the number of reviewers, the average number of reviews per capita is obtained, and based on the total number of reviews and the total review time, the average review efficiency per capita is obtained;
[0092] The marking efficiency, audit efficiency, re-examination efficiency, average number of markings per capita, average marking efficiency per capita, average number of audits per capita, average audit efficiency per capita, average number of re-examinations per capita, and average re-examination efficiency per capita are summarized to obtain the labor efficiency data.
[0093] Specifically, the number of annotators involved in the annotation, the number of auditors involved in the review, and the number of reviewers involved in the re-review are counted. According to the number of annotations for each annotated object, the annotation amount, review amount, and re-review amount completed by each annotated object are obtained, and the annotation amount, review amount, and re-review amount of all annotated objects are summarized to calculate the total annotation amount, the total review amount, and the total re-review amount of all annotated objects. According to the annotation time of each annotated object, the annotation time for each annotated object to complete the above annotation amount, the review time for completing the above review amount, and the re-review time for completing the above re-review amount are obtained, and the annotation time, review time, and re-review time of all annotated objects are summarized to calculate the total annotation time, the total review time, and the total re-review time of all annotated objects.
[0094] The annotation efficiency of each annotation object is obtained by dividing the annotation amount of each annotation object by the annotation time. The review efficiency of each annotation object is obtained by dividing the review amount of each annotation object by the review time. The review efficiency of the annotation object is obtained by dividing the review amount of each annotation object by the review time.
[0095] When the annotated object annotates pictures and videos, its identity is an annotator. When the annotated object reviews the annotated content, its identity is an auditor. When the annotated object reviews the annotated content, its identity is a reviewer. This embodiment defines: the average number of annotations per person is the average number of annotations per annotator within the range time; the average number of reviews per person is the average number of annotations per reviewer within the range time; the average number of reviews per person is the average number of annotations per reviewer within the range time; the average annotation efficiency per person is the average annotation efficiency per annotator within the range time; the average review efficiency per person is the average review efficiency per reviewer within the range time; the average review efficiency per person is the average review efficiency per reviewer within the range time.
[0096] Based on the above definition, the normalized formula for calculating the labor efficiency data for labeling tasks of different data types adopted in this embodiment is:
[0097] Per capita number of annotations = (total number of annotation results in the annotation tasks submitted by all personnel within the selected time) / (number of people involved in the annotation); Per capita number of reviews: (total number of annotation results in the review tasks submitted by all personnel within the selected time) / (number of people involved in the review); Per capita number of re-reviews: per capita number of re-reviews = (total number of annotation results selected in the review tasks submitted by all personnel within the selected time) / (number of people involved in the re-review); Per capita annotation efficiency: per capita annotation efficiency = (total number of annotation results marked by all personnel) / (total time taken by all personnel to mark); Per capita review efficiency: per capita review efficiency = (total number of annotation results reviewed by all personnel) / (total time taken by all personnel to review); Per capita review efficiency: per capita review efficiency = (total number of annotation results reviewed by all personnel) / (total time taken by all personnel to review).
[0098] In summary, the total number of annotations divided by the number of annotators can be used to calculate the average number of annotations per person. The total number of annotations divided by the total time taken to annotate can be used to calculate the average annotation efficiency. The total number of reviews divided by the number of reviewers can be used to calculate the average number of reviews per person. The total number of reviews divided by the total time taken to review can be used to calculate the average review efficiency. The total number of re-reviews divided by the number of reviewers can be used to calculate the average number of re-reviews per person. The total number of re-reviews divided by the total time taken to review can be used to calculate the average review efficiency. For example: from October 1st to October 3rd, the total number of re-reviews submitted by the reviewers is 100, and the number of reviewers participating in the re-review is 2. Then the average number of re-reviews per person = 100 / 2 = 50 / person, and the average review efficiency = 100 / (October 3rd - October 1st) = 50 / day.
[0099] The above-mentioned marking efficiency, audit efficiency, re-examination efficiency, average number of markings per capita, average number of audits per capita, average audit efficiency per capita, average number of re-examinations per capita and average re-examination efficiency are summarized to obtain the labor efficiency data.
[0100] The core calculation basis of this embodiment is to design a normalization formula based on time or the number of participants as the dimension, and use the normalization formula and the annotation quantity and annotation time stored in the database to output the data recorded in these databases through business calculations as a series of meaningful human efficiency data such as the average number of annotations per person of the annotators. Using a normalized calculation method can ensure that the same calculation logic and rules are used when calculating human efficiency data of different data types, thereby maintaining consistency. This can avoid differences in results due to the use of different methods and increase the reliability of data analysis.
[0101] In this embodiment, the labor efficiency data is calculated based on the number of annotations, the annotation time, and the labor efficiency data calculation formula, and a standard method is used to calculate the labor efficiency data of different data type annotation tasks. This makes the data format of the annotation platform adopt a unified standard, improves the consistency of the annotation platform, and reduces the complexity of development and maintenance. In addition, the efficiency of labor efficiency data calculation is improved.
[0102] In some optional implementations, determining the target annotation type corresponding to the annotated object according to the human efficiency data and the algorithm derivation function includes:
[0103] Based on the labor efficiency data, the labeling efficiency, audit efficiency and review efficiency of each labeling task completed by the labeling object are obtained;
[0104] According to the marking efficiency, the auditing efficiency, the reviewing efficiency and the algorithm derivation function, a first marking task corresponding to the maximum marking efficiency, a second marking task corresponding to the maximum reviewing efficiency and a third marking task corresponding to the maximum reviewing efficiency are obtained;
[0105] The data type of the first labeling task is used as the first target type, the data type of the second labeling task is used as the second target type, and the data type of the third labeling task is used as the third target type, wherein the first target type, the second target type and the third target type are included in the target labeling type.
[0106] Specifically, the labeling efficiency, auditing efficiency, and reviewing efficiency of the labeling object in completing each labeling task are read from the labor efficiency data.
[0107] This embodiment sets a feature value set {X 1 , X 2 , X 3 , ..., Xn}, where X represents the eigenvalue, X can be: the labeling efficiency, audit efficiency, and review efficiency of the labeling object for different data types of labeling tasks, and n is the total number of eigenvalues and the total number of types of labeling tasks. Based on the eigenvalue set, the algorithm derivation function of this embodiment is expressed in mathematical symbols as: Max(X 1 ,X 2 ,...,X n ), where Max represents the operation of finding the maximum value.
[0108] The above-calculated human efficiency data is input into the algorithm derivation function for processing. The algorithm derivation function can output the final result such as the target annotation type that the annotation object is good at according to a certain algorithm. The steps include: querying the annotation efficiency, audit efficiency and review efficiency of different annotation objects to complete each annotation task from the human efficiency data. According to the annotation efficiency, review efficiency, review efficiency and algorithm derivation function, the first annotation task corresponding to the maximum annotation efficiency, the second annotation task corresponding to the maximum review efficiency and the third annotation task corresponding to the maximum review efficiency are obtained. For example, there are 4 annotation tasks, and the data types are: face pictures, gesture pictures, parking space pictures, and dangerous driving. The review efficiency of the annotation object B for these 4 annotation tasks is: 500 / hour, 450 / hour, 600 / hour, and 300 / hour. At this time, n=4, X 1 =500,X 2 =450,X 3 =600,X 4 =300, Max(X 1 , X 2 , X 3 , X 4)=600, therefore, the second labeling task corresponding to the maximum review efficiency is the third labeling task. Similarly, the same steps can be used to obtain the first labeling task corresponding to the maximum labeling efficiency and the third labeling task corresponding to the maximum review efficiency. The data type of the first labeling task is taken as the first target type, the data type of the second labeling task is taken as the second target type, and the data type of the third labeling task is taken as the third target type. For example: if the data type of the third labeling task is parking space picture, then the second target type is parking space picture. Therefore, the labeling object B is good at reviewing parking space pictures. Similarly, the first target type and the third target type can be obtained. The above-mentioned first target type, second target type and third target type all belong to target labeling types.
[0109] The above-mentioned human efficiency data normalization calculation method and algorithm derivation function can support the scalability of the annotation page. When it is necessary to introduce new data types or update existing data types, the existing calculation method can be expanded without making large-scale changes to the entire calculation method.
[0110] In this embodiment, the algorithm derivation function derives the first labeling task corresponding to the maximum labeling efficiency, the second labeling task corresponding to the maximum review efficiency, and the third labeling task corresponding to the maximum review efficiency, and then obtains the first target type, the second target type, and the third target type, which is convenient for determining the data type suitable for labeling, the data type suitable for review, and the data type suitable for review for each labeling object, so as to facilitate more reasonable allocation of labeling tasks in the future.
[0111] In some optional implementations, after determining the target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function, the method further includes:
[0112] Display labor efficiency data and target annotation types on the target page.
[0113] Specifically, the target page is, for example, a page showing human efficiency data, such as Figure 2 As shown, the per capita annotation data, per capita audit data, per capita re-examination number, per capita annotation efficiency, per capita audit efficiency, and per capita re-examination efficiency in the labor efficiency data are displayed on the labor efficiency data display page. In addition, the number of annotations, annotation efficiency (pieces / hour), number of audits, audit efficiency (pieces / hour), number of re-examinations and re-examination efficiency (pieces / hour) of each annotation object, the data type of the annotation task, the statistical period of labor efficiency data, whether the annotation object is suitable for processing the annotation task of the current data type, and other information can be displayed separately. The difference between the labor efficiency data of each annotation object and the average value can also be calculated to determine the comprehensive level of the annotation object, for example: Figure 2The statistical period of the human efficiency data in is from July 5 to July 8. The data type of the annotation task is gesture recognition. The number of annotations of the annotation object C is 2411, which is 876 less than the average number of annotations per person of 3287, expressed as -876. The annotation efficiency is 241 (per hour), which is 231 less than the average annotation efficiency of 472 (per hour), expressed as -231. In addition, Figure 2 It can be seen that the annotation object E is marked with ※, indicating that the annotation object E is suitable for processing the annotation task of the current data type, that is, the annotation task of gesture recognition. For other contents, see Figure 2 , I will not go into details here.
[0114] In this embodiment, the labor efficiency data and the target annotation type are displayed on the target page, which is convenient for query and helps to determine the comprehensive level of each annotated object.
[0115] In this embodiment, a data annotation labor efficiency data calculation method is provided, which can be used in the above-mentioned computer device, can solve the same technical problems as steps S101 to S103, and has the same technical effects, such as Figure 3 As shown, the method comprises the following steps:
[0116] Step S301, the annotator enters the annotation page to annotate the task;
[0117] Step S302, when the annotator enters the task annotation page, the time t seconds is recorded. When the annotator completes the task, the time t+Δt seconds is recorded, and then the time Δt seconds of the task is saved in the database;
[0118] Step S303, when the annotator saves the task, the program will identify the image and save the number of annotations of the image to the database;
[0119] Step S304, the background outputs labor efficiency data through the labor efficiency calculation formula;
[0120] Step S305, processing these indicator data through an algorithm derivation function to obtain the corresponding final result;
[0121] Step S306, the final result is returned to the page end, and the page echoes the correct result.
[0122] For details, see Figure 1 Steps S101 to S103 of the illustrated embodiment and other embodiments described above will not be described in detail here.
[0123] In this embodiment, the use of background calculation of labor efficiency data greatly saves the time of the labelers, and the labor efficiency data also more clearly reflects the overall level of the labelers. The labor efficiency data is used to analyze each person's advantageous labeling type. The program can deduce which data types are suitable for each person to label based on the algorithm, so as to arrange the labeling tasks more reasonably. The method of normalizing the labor efficiency calculation of different labeling types can support the scalability of the system. When it is necessary to introduce new data types or update existing data types, the existing calculation method can be expanded without making large-scale changes to the entire calculation method.
[0124] In the present embodiment, a device for determining the target annotation type is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0125] This embodiment provides a device for determining a target annotation type. Figure 4 As shown, including:
[0126] An acquisition module 401 is used to acquire the number of annotations and the annotation time of the annotation objects when a first preset number of annotation objects complete the annotation task, wherein the data type of the annotation task has a second preset number;
[0127] Obtaining module 402, for obtaining the human efficiency data of the labeled object according to the number of labels, the labeling time and the human efficiency data calculation formula;
[0128] The determination module 403 is used to determine the target annotation type corresponding to the annotation object according to the human efficiency data and the algorithm derivation function, wherein the target annotation type is included in a second preset number of data types.
[0129] In some optional implementations, the acquisition module 401 includes:
[0130] A first acquisition unit is used to acquire the start time and the end time of the labeling task of the labeling object;
[0131] A second acquisition unit is used to acquire a first moment of suspending the annotation task and a second moment of resuming the annotation task if the annotation object suspends the annotation task during the process of processing the annotation task;
[0132] The first obtaining unit is used to obtain the marked time according to the start time, the end time, the first time and the second time.
[0133] In some optional implementations, the acquisition module 401 includes:
[0134] A third acquisition unit is used to acquire the information to be annotated in the annotation task;
[0135] The statistical unit is used to count the number of information to be labeled to obtain the number of labels.
[0136] In some optional implementations, the obtaining module 402 includes:
[0137] The second obtaining unit is used to obtain the number of annotators, the number of reviewers, and the number of re-reviewers according to the annotated object;
[0138] The third obtaining unit is used to obtain the annotation amount, the audit amount, the re-examination amount of each annotation object to complete the annotation task, the total annotation amount, the total audit amount and the total re-examination amount of all annotation objects according to the number of annotations;
[0139] The fourth obtaining unit is used to obtain the marking time, the review time, the re-examination time, the total marking time, the total review time and the total re-examination time of all the marking objects according to the marking time.
[0140] A fifth obtaining unit is used to obtain the labeling efficiency of the labeling object according to the labeling amount and the labeling time;
[0141] The sixth obtaining unit is used to obtain the audit efficiency of the marked object according to the audit quantity and the audit time;
[0142] A seventh obtaining unit is used to obtain the review efficiency of the marked object according to the review quantity and the review time;
[0143] An eighth obtaining unit is used to obtain the number of per capita annotations according to the total number of annotations and the number of annotation personnel, and to obtain the per capita annotation efficiency according to the total number of annotations and the total annotation time;
[0144] The ninth obtaining unit is used to obtain the number of per capita audits according to the total audit volume and the number of auditors, and to obtain the per capita audit efficiency according to the total audit volume and the total audit time;
[0145] The tenth obtaining unit is used to obtain the number of reviews per capita according to the total number of reviews and the number of reviewers, and to obtain the review efficiency per capita according to the total number of reviews and the total time spent on the reviews;
[0146] The summary unit is used to summarize the marking efficiency, audit efficiency, re-examination efficiency, the average number of markings per capita, the average number of audits per capita, the average audit efficiency per capita, the average number of re-examinations per capita and the average re-examination efficiency per capita to obtain labor efficiency data.
[0147] In some optional implementations, the determining module 403 includes:
[0148] The eleventh obtaining unit is used to obtain the labeling efficiency, the review efficiency and the re-examination efficiency of the labeling object for completing each labeling task according to the labor efficiency data;
[0149] A twelfth obtaining unit is used to obtain a first annotation task corresponding to the maximum annotation efficiency, a second annotation task corresponding to the maximum audit efficiency, and a third annotation task corresponding to the maximum review efficiency according to the annotation efficiency, the audit efficiency, the review efficiency, and the algorithm derivation function;
[0150] A setting unit is used to set the data type of the first labeling task as the first target type, the data type of the second labeling task as the second target type, and the data type of the third labeling task as the third target type, wherein the first target type, the second target type and the third target type are included in the target labeling type.
[0151] In some optional embodiments, the device further comprises:
[0152] The display module is used to display the labor efficiency data and target annotation type on the target page.
[0153] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0154] The target annotation type determination device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0155] The embodiment of the present invention also provides a computer device having the above Figure 4 The target labeling type determination device shown.
[0156] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.
[0157] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0158] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0159] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0160] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0161] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0162] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0163] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0164] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for determining a target annotation type, characterized in that: The method comprises: When a first preset number of labeled objects complete a labeling task, obtaining the number of labels and the labeling time of the labeled objects, wherein the data type of the labeling task has a second preset number; Obtaining the annotation time of the annotation object includes: obtaining the start time and the end time of the annotation object processing the annotation task; if the annotation object suspends the annotation task during the processing of the annotation task, obtaining a first time of suspending the annotation task and a second time of resuming the execution of the annotation task; obtaining the annotation time according to the start time, the end time, the first time and the second time; Obtaining the human efficiency data of the marked object according to the number of markings, the marking time and a human efficiency data calculation formula; The human efficiency data of the labeled object is obtained according to the number of labels, the labeling time and the human efficiency data calculation formula, including: according to the labeled object, the number of labelers, the number of auditors and the number of re-reviewers are obtained; according to the number of labels, the labeling amount, the audit amount, the re-review amount of each labeled object to complete the labeling task, the total labeling amount, the total audit amount and the total re-review amount of all labeled objects are obtained; according to the labeling time, the labeling time, the re-review time of each labeled object to complete the labeling task, the total labeling time, the total audit time and the total re-review time of all labeled objects are obtained; according to the labeling amount and the labeling time, the labeling efficiency of the labeled object is obtained; according to the audit amount and the re-review time, the re-review efficiency of the labeled object is obtained; according to the re-review According to the total number of reviews and the number of reviewers, the review efficiency of the marked object is obtained; according to the total number of marks and the number of markers, the average number of marks per capita is obtained, and according to the total number of marks and the total time for marking, the average marking efficiency is obtained; according to the total number of reviews and the number of reviewers, the average number of reviews per capita is obtained, and according to the total number of reviews and the total time for review, the average review efficiency is obtained; according to the total number of reviews and the number of reviewers, the average number of reviews per capita is obtained, and according to the total number of reviews and the total time for review, the average review efficiency is obtained; the marking efficiency, the review efficiency, the average number of marks per capita, the average marking efficiency, the average number of reviews per capita, the average review efficiency, the average number of reviews per capita and the average review efficiency are summarized to obtain the labor efficiency data; The labor efficiency data calculation formula includes: average number of annotations per person = (total number of annotation results in the annotation tasks submitted by all personnel within the selected time) / (number of people involved in the annotation); average number of reviews per person = (total number of annotation results in the review tasks submitted by all personnel within the selected time) / (number of people involved in the review); average number of re-reviews per person = (total number of annotation results selected in the re-review tasks submitted by all personnel within the selected time) / (number of people involved in the re-review); average annotation efficiency per person = (total number of annotation results marked by all personnel) / (total annotation time of all personnel); average review efficiency per person = (total number of annotation results reviewed by all personnel) / (total review time of all personnel); average review efficiency per person = (total number of annotation results reviewed by all personnel) / (total review time of all personnel); According to the human efficiency data and the algorithm derivation function, a target annotation type corresponding to the annotation object is determined, wherein the target annotation type is included in a preset number of the data types.
2. The method according to claim 1, characterized in that Obtaining the number of annotations of the annotated object includes: Obtaining the information to be annotated in the annotation task; The number of the information to be annotated is counted to obtain the number of annotations.
3. The method according to claim 1, characterized in that The determining, according to the human efficiency data and the algorithm derivation function, the target annotation type corresponding to the annotation object includes: According to the labor efficiency data, the labeling efficiency, the auditing efficiency and the reviewing efficiency of the labeling object in completing each labeling task are obtained; According to the labeling efficiency, the auditing efficiency, the review efficiency and the algorithm derivation function, a first labeling task corresponding to the maximum labeling efficiency, a second labeling task corresponding to the maximum review efficiency and a third labeling task corresponding to the maximum review efficiency are obtained; The data type of the first labeling task is used as the first target type, the data type of the second labeling task is used as the second target type, and the data type of the third labeling task is used as the third target type, wherein the first target type, the second target type and the third target type are included in the target labeling type.
4. The method according to claim 1, characterized in that: After determining the target annotation type corresponding to the annotated object according to the human efficiency data and the algorithm derivation function, the method further includes: The labor efficiency data and the target annotation type are displayed on the target page.
5. A device for determining a target annotation type, characterized in that: The device comprises: An acquisition module, configured to acquire the number of annotations and the annotation time of the annotated objects when the annotated objects of a first preset number complete the annotation task, wherein the data type of the annotated task has a second preset number; The acquisition module includes: a first acquisition unit, used to acquire the start time and the end time of the annotation object processing the annotation task; a second acquisition unit, used to acquire the first time of suspending the annotation task and the second time of resuming the annotation task if the annotation object suspends the annotation task during the process of processing the annotation task; a first obtaining unit, used to obtain the annotation time according to the start time, the end time, the first time and the second time; An obtaining module, used for obtaining the human efficiency data of the annotated object according to the number of annotations, the annotation time and the human efficiency data calculation formula; The obtaining module includes: a second obtaining unit, which is used to obtain the number of annotators, the number of auditors, and the number of re-reviewers according to the annotated object; a third obtaining unit, which is used to obtain the annotation amount, the audit amount, the re-review amount of each annotated object to complete the annotation task, the total annotation amount, the total audit amount, and the total re-review amount of all annotated objects according to the number of annotations; a fourth obtaining unit, which is used to obtain the annotation time, the audit time, the re-review time of each annotated object to complete the annotation task, the total annotation time, the total audit time, and the total re-review time of all annotated objects according to the annotation time; a fifth obtaining unit, which is used to obtain the annotation efficiency of the annotated object according to the annotation amount and the annotation time; a sixth obtaining unit, which is used to obtain the re-review efficiency of the annotated object according to the re-review amount and the re-review time; and a seventh obtaining unit, which is used to obtain the total annotation time, the total audit time, and the total re-review time of each annotated object to complete the annotation task. The eighth obtaining unit is used to obtain the average number of annotations according to the total number of annotations and the number of annotation personnel, and to obtain the average annotation efficiency according to the total number of annotations and the total annotation time; the ninth obtaining unit is used to obtain the average number of audits according to the total number of audits and the number of auditors, and to obtain the average audit efficiency according to the total number of audits and the total audit time; the tenth obtaining unit is used to obtain the average number of re-examinations according to the total number of re-examinations and the number of re-examination personnel, and to obtain the average re-examination efficiency according to the total number of re-examinations and the total re-examination time; the summarizing unit is used to summarize the annotation efficiency, the audit efficiency, the re-examination efficiency, the average number of annotations, the average annotation efficiency, the average number of audits, the average audit efficiency, the average number of re-examinations and the average re-examination efficiency to obtain the human efficiency data; The labor efficiency data calculation formula includes: average number of annotations per person = (total number of annotation results in the annotation tasks submitted by all personnel within the selected time) / (number of people involved in the annotation); average number of reviews per person = (total number of annotation results in the review tasks submitted by all personnel within the selected time) / (number of people involved in the review); average number of re-reviews per person = (total number of annotation results selected in the re-review tasks submitted by all personnel within the selected time) / (number of people involved in the re-review); average annotation efficiency per person = (total number of annotation results marked by all personnel) / (total annotation time of all personnel); average review efficiency per person = (total number of annotation results reviewed by all personnel) / (total review time of all personnel); average review efficiency per person = (total number of annotation results reviewed by all personnel) / (total review time of all personnel); A determination module is used to determine a target annotation type corresponding to the annotation object according to the human efficiency data and an algorithm derivation function, wherein the target annotation type is included in a second preset number of the data types.
6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the target labeling type determination method according to any one of claims 1 to 4 by executing the computer instructions.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the target labeling type determination method according to any one of claims 1 to 4.
8. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the target labeling type determination method according to any one of claims 1 to 4.
