A method, device, electronic device and readable storage medium for reviewing annotation data

Abstract

The present invention provides a method, device, electronic device and readable storage medium for reviewing annotated data, and relates to the field of data processing technology. The method comprises: determining the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed; determining the first sampling file data in the first batch according to the sampling ratio, and reviewing the first sampling file data; and determining whether to conduct a second batch review when the first sampling file data has been reviewed. The solution of the present invention solves the problem that batch data belonging to the same data set cannot be continuously viewed during the review of annotated data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and readable storage medium for reviewing labeled data. Background Art

[0002] At present, in order to train the algorithm models related to autonomous driving, the annotation platform provides vehicle data for annotation. There are three concepts in the annotation platform: project, data set and batch. Among them, a project will be associated with multiple data sets, a data set will be associated with multiple continuous frames of pictures, a project will be associated with multiple batch tasks, and a single batch may belong to data set A or data set B. The annotator annotates the image data in batches. After annotating a project, many batches will be generated. Then the auditor will receive the batch task according to the batch creation time and review the accuracy of the data annotation. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device, electronic device and readable storage medium for reviewing labeled data, so as to solve the problem in the prior art that batch data belonging to the same data set cannot be reviewed continuously during the review of labeled data.

[0004] An embodiment of the present invention provides a method for reviewing labeled data, including:

[0005] Determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed;

[0006] Determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and review the first sampling inspection file data;

[0007] When the first sampling inspection file data have been reviewed, determine whether to conduct a second batch review.

[0008] Optionally, in the method for reviewing annotated data, before determining the batch information of the target data set associated with the item to be reviewed, the method further includes:

[0009] The target data set is determined according to the following steps:

[0010] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0011] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0012] Optionally, the method for reviewing annotated data, before determining the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed, the method further includes:

[0013] Get the review status of the historical batches of the project to be reviewed.

[0014] Optionally, the method for reviewing the labeled data, determining the first sampling file data in the first batch according to the sampling ratio, includes:

[0015] Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data;

[0016] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

[0017] Optionally, the method for reviewing the labeled data, after determining the first sampling file data in the first batch according to the sampling ratio, the method further comprises:

[0018] Updates the number of reviewed batches for the target dataset.

[0019] Optionally, the method for reviewing the labeled data, after determining the first sampling file data in the first batch according to the sampling ratio, the method further comprises:

[0020] Update the name of the first batch.

[0021] Optionally, the method for reviewing the annotated data further comprises:

[0022] In the case where it is determined that the second batch is to be reviewed, the second sampling document data in the second batch is determined according to the sampling ratio, and the second sampling document data is reviewed.

[0023] An embodiment of the present invention further provides an electronic device, including a processor and a transceiver, wherein:

[0024] The processor is used to determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed;

[0025] The processor is further configured to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data;

[0026] The processor is further configured to determine whether to conduct a second batch review when the review of the first sampling file data has been completed.

[0027] Optionally, in the electronic device, the processor is further configured to:

[0028] The target data set is determined according to the following steps:

[0029] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0030] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0031] Optionally, in the electronic device, the processor is further configured to:

[0032] Get the review status of the historical batches of the project to be reviewed.

[0033] Optionally, in the electronic device, the processor is specifically configured to:

[0034] Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data;

[0035] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

[0036] Optionally, in the electronic device, the processor is further configured to:

[0037] Updates the number of reviewed batches for the target dataset.

[0038] Optionally, in the electronic device, the processor is further configured to:

[0039] Update the name of the first batch.

[0040] Optionally, in the electronic device, the processor is further configured to:

[0041] In the case where it is determined that the second batch is to be reviewed, the second sampling document data in the second batch is determined according to the sampling ratio, and the second sampling document data is reviewed.

[0042] The embodiment of the present invention further provides a labeling data review device, comprising:

[0043] A first determination module, used to determine the sampling ratio and batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed;

[0044] A second determination module is used to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data;

[0045] The judgment module is used to judge whether to conduct a second batch of review when the review of the first sampling file data has been completed.

[0046] Optionally, the device for reviewing the labeled data further comprises:

[0047] The third determination module is used to determine the target data set according to the following steps:

[0048] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0049] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0050] Optionally, the device for reviewing the labeled data further comprises:

[0051] The acquisition module is used to obtain the review status of the historical batches of the items to be reviewed.

[0052] Optionally, in the annotated data review device, the second determination module is specifically configured to:

[0053] Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data;

[0054] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

[0055] Optionally, the device for reviewing the labeled data further comprises:

[0056] The first updating module is used to update the number of reviewed batches of the target data set.

[0057] Optionally, the device for reviewing the labeled data further comprises:

[0058] The second updating module is used to update the name of the first batch.

[0059] Optionally, the device for reviewing the labeled data further comprises:

[0060] The fourth determination module is used to determine the second sampling file data in the second batch according to the sampling ratio when it is determined that the second batch review is to be conducted, and to review the second sampling file data.

[0061] An embodiment of the present invention also provides an electronic device, comprising: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the labeling data review method as described in any one of the above items is implemented.

[0062] An embodiment of the present invention further provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the method for reviewing the labeled data as described in any one of the above items is implemented.

[0063] The beneficial effects of the above technical solution of the present invention are as follows:

[0064] The embodiment of the present invention determines the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed; determines the first sampling file data in the first batch according to the sampling ratio, and reviews the first sampling file data; when the first sampling file data have completed the review, determines whether to conduct a second batch review, simplifies the review process, improves the accuracy, and can automatically jump to the next batch after reviewing one batch, realizing continuous frame review data, greatly improving the review efficiency, and there is no need to re-enter the sampling ratio after reviewing each batch. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of the steps of a method for reviewing labeled data according to an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of a specific process of the method for reviewing labeled data according to an embodiment of the present invention;

[0067] Figure 3 This is one of the structural schematic diagrams of an electronic device according to an embodiment of the present invention;

[0068] Figure 4 A schematic diagram of a labeling data review device according to an embodiment of the present invention;

[0069] Figure 5 This is a second structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0071] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0072] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0073] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0074] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0075] like Figure 1 As shown, an embodiment of the present invention provides a method for reviewing labeled data, including:

[0076] Step S101: Determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed.

[0077] In actual application of the embodiment of the present invention, the user can select a project name (or a project identity) in the review list interface as the project to be reviewed, and obtain the project configuration information according to the name of the project to be reviewed.

[0078] It should be noted that each project is associated with at least one data set, and each project is associated with at least one batch. Different batches may belong to the same data set or to different data sets.

[0079] Step S102: Determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and review the first sampling inspection file data.

[0080] In an embodiment of the present invention, according to the batch information, all batch tasks belonging to the target data set are obtained, including the first batch, the second batch, the third batch, etc.; according to the batch information, the data quantity and file name data of each batch can also be obtained.

[0081] Specifically, determining the first sampling inspection file data in the first batch according to the sampling inspection ratio includes:

[0082] The quantity of the first sampling file data is determined according to the sampling ratio (qc_shot) and the quantity of the first batch of data (total). Here, the quantity of the first sampling file data is obtained by rounding up the remainder of qc_shot / 100*total.

[0083] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved. Here, the number of files of the first sampling file data is randomly selected from the file name data of the first batch to determine the file name (file_id) of the first sampling file data, and the file name (file_id) of the first sampling file data is saved in the (res_data) field.

[0084] Step S103: When the first sampling inspection file data have all been reviewed, determine whether to conduct a second batch review.

[0085] In an embodiment of the present invention, when it is determined that the second batch is to be reviewed, step S102 is repeated for the second batch, that is, the second sampling file data in the second batch is determined according to the sampling ratio, and the second sampling file data is reviewed.

[0086] Furthermore, when the second sampling file data has completed the review, it is determined whether to conduct a third batch review until there are no batches to be reviewed under the target data set, thereby simplifying the review process. The review can automatically jump to the next batch to achieve continuous frame review, thereby improving the review efficiency and accuracy, and there is no need to re-enter the sampling ratio after each batch is reviewed.

[0087] It should be noted that if it is determined that the second batch of reviews will not be conducted, the interface will be returned to the review list.

[0088] The embodiment of the present invention determines the sampling ratio and the batch information of the target data set associated with the project to be audited according to the project configuration information of the project to be audited; determines the first sampling file data in the first batch according to the sampling ratio, and audits the first sampling file data; and determines whether to conduct a second batch audit when the audit of the first sampling file data has been completed, thereby simplifying the audit process, improving the audit efficiency and accuracy, and can automatically jump to the next batch after auditing one batch, and can continuously audit data, greatly improving the audit efficiency, and there is no need to re-enter the sampling ratio after auditing each batch.

[0089] Optionally, in the method for reviewing annotated data, before determining the batch information of the target data set associated with the item to be reviewed, the method further includes:

[0090] The target data set is determined according to the following steps:

[0091] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0092] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0093] In an embodiment of the present invention, if the data priority information in the project configuration information is obtained, at least one data set associated with the project to be reviewed is prioritized according to the priority information, and the data set with the highest priority is selected as the target data set, wherein the data priority information includes the name and version number of the data set. Here, the data priority information can be adjusted so as to give priority to the urgent data in the project to be reviewed.

[0094] If the data priority information in the project configuration information is not obtained, the target data set is directly determined according to the data selection information input by the user.

[0095] Optionally, the method for reviewing annotated data, before determining the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed, the method further includes:

[0096] Get the review status of the historical batches of the project to be reviewed.

[0097] It should be noted that after determining the name of the project to be reviewed, the review status of the historical batches of the project to be reviewed is obtained to find out whether there are batch tasks that meet the status of pending review, that is, batch tasks that have not been reviewed and completed. If so, the user is prompted to review and complete the received batch tasks first. If not, execute step S101.

[0098] Optionally, the method for reviewing the labeled data, after determining the first sampling file data in the first batch according to the sampling ratio, the method further comprises:

[0099] Updates the number of reviewed batches for the target dataset.

[0100] In the embodiment of the present invention, the number of reviewed batches of the target data set is updated according to the name of the project to be reviewed, the name and the version number of the target data set, that is, the number of reviewed batches is increased once.

[0101] Optionally, the method for reviewing the labeled data, after determining the first sampling file data in the first batch according to the sampling ratio, the method further comprises:

[0102] Update the name of the first batch.

[0103] In the embodiment of the present invention, the name of the first batch is updated to the name of the current user, and the reviewer of each batch of tasks is clarified. Moreover, after the user receives the batch task, he can enter the "My Task List" to start reviewing the data.

[0104] Combine the following Figure 2 , specifically explain the annotation data review process:

[0105] Step S201: The user selects the name of the item to be reviewed in the review list interface.

[0106] Step S202: Acquire the review status of the historical batches of the user's pending review project, and determine whether any received batch tasks under the user's pending review project have an unfinished review status.

[0107] If the judgment result of step S202 is yes, then enter step S203: prompt the user to review the received batch tasks in the unfinished review status first, and exit the process.

[0108] If the judgment result of step S202 is no, then proceed to step S204: determine whether there is data priority information in the project configuration information of the project to be reviewed.

[0109] If the judgment result of step S204 is yes, then proceed to step S205: determine the target data set associated with the project to be reviewed according to the data priority information.

[0110] If the judgment result of step S204 is no, then proceed to step S206: determine the target data set associated with the project to be reviewed according to the data selection information input by the user, wherein the data selection information includes the name of the data set and the version of the data set.

[0111] Step S207: Determine the sampling ratio and batch information of the target data set according to the project configuration information, wherein the batch information includes all batch tasks belonging to the target data set, the data quantity and file name data of each batch.

[0112] Step S208: Determine the quantity of the first sampling file data according to the sampling ratio and the quantity of the first batch of data.

[0113] Step S209: Determine the file name of the first sampled file data according to the quantity of the first sampled file data and the file name data of the first batch, and save the file name of the first sampled file data.

[0114] Step S210: Update the number of reviewed batches of the target data set.

[0115] Step S211: Update the name of the first batch according to the user's name.

[0116] Step S212: Start reviewing the first sampling file data.

[0117] Step S213: When the first batch of sample inspection documents have been reviewed, determine whether to enter the second batch review.

[0118] If the judgment result of step S213 is yes, then re-enter step S214: determine the amount of the second sampling file data according to the sampling ratio and the amount of data in the second batch.

[0119] If the judgment result of step S213 is no, then exit the process.

[0120] It should be noted that the embodiments of the present invention can reduce the complexity of the user's review operation. The user only needs to click once to receive the task to complete the review of the entire target data set. In addition, the accuracy of the review results is improved through the continuous frame display of the data, which makes it easier to discover erroneous data, prevent the omission of erroneous data, and avoid reducing the accuracy of the labeled data.

[0121] The embodiment of the present invention further provides an electronic device 300, including a processor 301 and a transceiver 302, wherein:

[0122] The processor 301 is used to determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed;

[0123] The processor 301 is further configured to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data;

[0124] The processor 301 is further configured to determine whether to conduct a second batch review when the review of the first sampling file data has been completed.

[0125] The embodiment of the present invention determines the sampling ratio and the batch information of the target data set associated with the project to be audited according to the project configuration information of the project to be audited; determines the first sampling file data in the first batch according to the sampling ratio, and audits the first sampling file data; when the audit of the first sampling file data is completed, determines whether to conduct a second batch audit, simplifies the audit process, improves the audit efficiency and accuracy, can automatically jump to the next batch after auditing one batch, can continuously audit data, greatly improves the audit efficiency, and does not need to re-enter the sampling ratio after auditing each batch.

[0126] Optionally, in the electronic device 300, the processor 301 is further configured to:

[0127] The target data set is determined according to the following steps:

[0128] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0129] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0130] Optionally, in the electronic device 300, the processor 301 is further configured to:

[0131] Get the review status of the historical batches of the project to be reviewed.

[0132] Optionally, in the electronic device 300, the processor 301 is specifically configured to:

[0133] Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data;

[0134] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

[0135] Optionally, in the electronic device 300, the processor 301 is further configured to:

[0136] Updates the number of reviewed batches for the target dataset.

[0137] Optionally, in the electronic device 300, the processor 301 is further configured to:

[0138] Update the name of the first batch.

[0139] Optionally, in the electronic device 300, the processor 301 is further configured to:

[0140] In the case where it is determined that the second batch is to be reviewed, the second sampling document data in the second batch is determined according to the sampling ratio, and the second sampling document data is reviewed.

[0141] The embodiment of the present invention further provides a labeling data review device, comprising:

[0142] The first determination module 401 is used to determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed;

[0143] A second determination module 402 is used to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data;

[0144] The judgment module 403 is used to judge whether to conduct a second batch of review when the review of the first sampling file data has been completed.

[0145] The embodiment of the present invention determines the sampling ratio and the batch information of the target data set associated with the project to be audited according to the project configuration information of the project to be audited; determines the first sampling file data in the first batch according to the sampling ratio, and audits the first sampling file data; when the audit of the first sampling file data is completed, determines whether to conduct a second batch audit, simplifies the audit process, improves the audit efficiency and accuracy, can automatically jump to the next batch after auditing one batch, can continuously audit data, greatly improves the audit efficiency, and does not need to re-enter the sampling ratio after auditing each batch.

[0146] Optionally, the device for reviewing the labeled data further comprises:

[0147] The third determination module is used to determine the target data set according to the following steps:

[0148] In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information;

[0149] In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

[0150] Optionally, the device for reviewing the labeled data further comprises:

[0151] The acquisition module is used to obtain the review status of the historical batches of the items to be reviewed.

[0152] Optionally, in the annotated data review device, the second determination module 402 is specifically configured to:

[0153] Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data;

[0154] According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

[0155] Optionally, the device for reviewing the labeled data further comprises:

[0156] The first updating module is used to update the number of reviewed batches of the target data set.

[0157] Optionally, the device for reviewing the labeled data further comprises:

[0158] The second updating module is used to update the name of the first batch.

[0159] Optionally, the device for reviewing the labeled data further comprises:

[0160] The fourth determination module is used to determine the second sampling file data in the second batch according to the sampling ratio when it is determined that the second batch review is to be conducted, and to review the second sampling file data.

[0161] It should be noted that the above-mentioned device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned annotation data review method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0162] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes: a processor 501; and a memory 503 connected to the processor 501 through a bus interface 502, the memory 503 is used to store programs and data used by the processor 501 when performing operations, and the processor 501 calls and executes the programs and data stored in the memory 503.

[0163] The transceiver 504 is connected to the bus interface 502 and is used to receive and send data under the control of the processor 501 .

[0164] Among them, Figure 5 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 501 and memory represented by memory 503. The bus architecture may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are not further described herein. The bus interface 502 provides an interface for the user, such as Figure 5 The user interface 505 in the embodiment of the present invention. The transceiver 504 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 501 is responsible for managing the bus architecture and general processing, and the memory 503 may store data used by the processor 501 when performing operations.

[0165] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing related hardware through a program, wherein the program includes instructions for executing part or all of the steps of the above method; and the program may be stored in a readable storage medium, which may be any form of storage medium.

[0166] An embodiment of the present invention further provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the method for reviewing the labeled data as described in any one of the above items is implemented.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0169] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0170] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles described in the present invention. These improvements and modifications are also within the scope of protection of the present invention.

Claims

1. A method for reviewing labeled data, characterized in that: include: Determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed; wherein each project is associated with at least one data set, and each project is associated with at least one batch, and different batches may belong to the same data set or to different data sets; According to the batch information, all batch tasks belonging to the target data set are obtained, including at least the first batch and the second batch; Determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and review the first sampling inspection file data; When the first sampling inspection file data have been reviewed, determine whether to conduct a second batch review; In the case where it is determined that the second batch is to be reviewed, the second sampling document data in the second batch is determined according to the sampling ratio, and the second sampling document data is reviewed.

2. The method for reviewing labeled data according to claim 1, characterized in that: Before determining the batch information of the target data set associated with the item to be reviewed, the method further includes: The target data set is determined according to the following steps: In case that the data priority information in the project configuration information is obtained, determining the target data set according to the data priority information; In the case where the data priority information is not obtained, the target data set is determined according to the data selection information input by the user.

3. The method for reviewing annotated data according to claim 1, characterized in that: Before determining the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed, the method further includes: Get the review status of the historical batches of the project to be reviewed.

4. The method for reviewing annotated data according to claim 1, characterized in that: Determining the first sampling inspection file data in the first batch according to the sampling inspection ratio includes: Determining the quantity of the first sampling inspection file data according to the sampling inspection ratio and the quantity of the first batch of data; According to the quantity of the first sampling file data and the file name data of the first batch, the file name of the first sampling file data is determined, and the file name of the first sampling file data is saved.

5. The method for reviewing annotated data according to claim 1, characterized in that: After determining the first sampling inspection file data in the first batch according to the sampling inspection ratio, the method further includes: Updates the number of reviewed batches for the target dataset.

6. The method for reviewing annotated data according to claim 1, characterized in that: After determining the first sampling inspection file data in the first batch according to the sampling inspection ratio, the method further includes: Update the name of the first batch.

7. An electronic device, comprising a processor and a transceiver, characterized in that: in: The processor is used to determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed; wherein each project is associated with at least one data set, and each project is associated with at least one batch, and different batches may belong to the same data set or to different data sets; The processor is further configured to obtain, according to the batch information, all batch tasks belonging to the target data set, including at least a first batch and a second batch; The processor is further configured to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data; The processor is further configured to, when the first sampling inspection file data have all been reviewed, determine whether to conduct a second batch review; The processor is further configured to, when it is determined that the second batch review is to be conducted, determine the second sampling inspection file data in the second batch according to the sampling inspection ratio, and review the second sampling inspection file data.

8. A labeling data review device, characterized in that: include: A first determination module is used to determine the sampling ratio and the batch information of the target data set associated with the project to be reviewed according to the project configuration information of the project to be reviewed; wherein each project is associated with at least one data set, and each project is associated with at least one batch, and different batches may belong to the same data set or to different data sets; The first determination module is further used to obtain all batch tasks belonging to the target data set according to the batch information, including at least the first batch and the second batch; A second determination module is used to determine the first sampling inspection file data in the first batch according to the sampling inspection ratio, and to review the first sampling inspection file data; A judgment module, used to judge whether to conduct a second batch of review when the review of the first sampling inspection file data has been completed; The fourth determination module is used to determine the second sampling file data in the second batch according to the sampling ratio when it is determined that the second batch review is to be conducted, and to review the second sampling file data.

9. An electronic device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the labeling data review method as described in any one of claims 1 to 6 is implemented.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the annotation data review method according to any one of claims 1 to 6 is implemented.

Images