Semi-automatic labeling method for improving labeling precision and efficiency and related device

Through the semi-automatic labeling method, combined with data preprocessing, automatic labeling engine and human-computer interactive interface, the problems of low efficiency, high cost and difficult consistency of traditional data labeling methods are solved, efficient and accurate labeling results are achieved, and automatic optimization is optimized. The labeling engine continuously improves labeling efficiency and accuracy.

CN118964888AInactive Publication Date: 2024-11-15GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)

Patent Information

Application Number
CN202411016754.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional data annotation methods are inefficient, expensive, difficult to ensure consistency and accuracy, and difficult to scale and adapt to advanced annotation tasks. They lack feedback loops and quality control, especially in large-scale and multimodal data processing.

Method used

The semi-automatic labeling method is adopted to expand through data preprocessing, initial labeling of automatic labeling engines, human-computer interactive interface correction and quality control, combined with customized requirements, to form customized labeling data, reduce manual intervention and improve labeling speed and accuracy.

Benefits of technology

It significantly improves the labeling speed and accuracy, reduces costs, ensures the consistency and quality of the labeling results, and continuously improves the labeling efficiency and accuracy through the optimization of the automatic labeling engine, forming a virtuous cycle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964888A_ABST
    Figure CN118964888A_ABST
Patent Text Reader

Abstract

The invention discloses a semi-automatic labeling method for improving labeling precision and efficiency and a related device, and the method comprises the steps: obtaining to-be-labeled data, carrying out the data preprocessing of the to-be-labeled data, and forming the preprocessed to-be-labeled data; performing preliminary annotation processing on the preprocessed to-be-annotated data based on an automatic annotation engine to obtain preliminary annotation data; rendering the preliminary annotation data into a human-computer interaction interface for display processing, and carrying out annotation correction processing by an annotator based on the displayed preliminary annotation data to obtain corrected annotation data; annotation quality control and auditing processing are conducted on the corrected annotation data according to a preset rule, and audited annotation data are obtained; and performing expansion and customization processing on the audited annotation data based on a customization demand to form customized annotation data. In the embodiment of the invention, the overall speed of providing data annotation can be realized, and the data annotation cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a semi-automatic labeling method and related devices for improving labeling accuracy and efficiency. Background Art

[0002] Anomaly detection plays an important role in the industrial field. With the rapid development of big data and the Internet of Things, the amount of data generated in industrial systems is also increasing. Traditional labeling methods have some bottlenecks in large-scale data processing. The cumbersome manual labeling is time-consuming and labor-intensive, inefficient, and difficult to ensure the consistency and accuracy of the labeling. The traditional labeling method requires manual concentration and participation. Long-term repetitive work often makes the labelers feel tired and bored, which easily leads to a decline in work quality.

[0003] Traditional data labeling methods refer to data classification and labeling processes that mainly rely on manual work. These methods played a key role in the early development of artificial intelligence and machine learning, but with the advancement of technology and the explosive growth of data size, they gradually exposed a series of limitations and challenges. The following are several major problems with traditional data labeling methods: High cost: Manual labeling is labor-intensive and requires a lot of time and manpower. As the data set increases, the cost rises proportionally, which may become an unbearable burden for large-scale projects; Low efficiency: Compared with automated or semi-automated labeling techniques, manual inspection and labeling of data one by one is much slower. Especially when dealing with high-dimensional or continuous streaming data (such as video, audio), the efficiency problem is more prominent; Consistency problem: There may be differences in understanding and operation between different labelers, resulting in inconsistent labeling of the same type of data, affecting the quality of model training and the accuracy of the final model; Fluctuation of labeling quality: People have limited attention and endurance, and long-term monotonous labeling work can easily lead to fatigue, thus affecting the accuracy and meticulousness of labeling. In addition, the lack of an effective quality control mechanism will also exacerbate this problem; difficulty in scalability: Faced with large-scale, multimodal data sets, traditional methods are difficult to respond and process quickly, limiting the expansion and deepening of AI applications; poor technical adaptability: With the development of AI technology, the complexity and diversity of data are increasing, and traditional methods are often difficult to efficiently handle advanced labeling tasks such as semantic understanding and sentiment analysis; lack of feedback loop: The manual labeling process is usually one-time, lacking feedback linkage with subsequent model training results, and unable to dynamically optimize labeling standards or improve labeling efficiency; professional knowledge requirements: For data in specific fields (such as medical images, legal documents), high-quality labeling requires professional background knowledge, which limits the available labor pool and further increases costs; management difficulties: large-scale labeling projects involve the coordination and management of many labelers, and traditional methods have difficulties in task allocation, progress tracking, quality monitoring, etc. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a semi-automatic labeling method and related devices for improving labeling accuracy and efficiency, which can provide the overall speed of data labeling and reduce the cost of data labeling.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a semi-automatic annotation method for improving annotation accuracy and efficiency, the method comprising:

[0006] Obtaining data to be labeled, and performing data preprocessing on the data to be labeled to form preprocessed data to be labeled;

[0007] Perform preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data;

[0008] Rendering the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data;

[0009] Performing labeling quality control and review processing on the revised labeling data according to preset rules to obtain reviewed labeling data;

[0010] Based on customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

[0011] Optionally, the performing data preprocessing on the data to be labeled to form preprocessed data to be labeled includes:

[0012] Performing format conversion processing on the data to be annotated, converting the data to be annotated from different sources into data to be annotated in a unified format;

[0013] Perform data cleaning and standardization on the data to be labeled in a unified format to obtain standardized data to be labeled;

[0014] A preliminary analysis is performed on the standardized data to be labeled, and feature data helpful for labeling is extracted based on the preliminary analysis results to form preprocessed data to be labeled.

[0015] Optionally, performing preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data includes:

[0016] The preprocessed data to be labeled is input into the automatic labeling engine for labeling according to the data type and label, and the data labeling result is evaluated for execution to generate preliminary labeled data. The automatic labeling engine is generated by inputting historical data into the deep learning model for training after labeling according to the data type and requirements.

[0017] Optionally, rendering the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performing annotation correction processing based on the displayed preliminary annotation data to obtain the corrected annotation data, includes:

[0018] Rendering the preliminary marked data according to the target object marked by the marked box, point or line to obtain a rendering result;

[0019] Loading the rendering result into the human-computer interaction interface for display processing;

[0020] The annotator performs manual annotation correction processing according to the preliminary annotation data displayed on the human-computer interaction interface to obtain corrected annotation data.

[0021] Optionally, after obtaining the corrected annotation data, the method further includes:

[0022] The corrected annotation data is returned to the automatic annotation engine, and the automatic annotation engine is subjected to correction processing of node parameters.

[0023] Optionally, the performing labeling quality control and review processing on the corrected labeling data according to preset rules to obtain the reviewed labeling data includes:

[0024] Using an automatic annotation checking algorithm to check the logical errors and inconsistencies in the corrected annotation data, and obtaining the number of logical errors and inconsistencies in the corrected annotation data;

[0025] When the number of logical errors and inconsistencies in the corrected annotation data is greater than a preset number, the corrected annotation data is put into a manual review process, and an expert verification process is performed in the manual review process to form reviewed annotation data.

[0026] Optionally, the reviewed annotation data is expanded and customized based on the customization requirements to form customized annotation data, including:

[0027] Based on the data format and annotation rules in the customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

[0028] In addition, an embodiment of the present invention further provides a semi-automatic annotation method for improving annotation accuracy and efficiency, the method comprising:

[0029] Data preprocessing module: used to obtain the data to be labeled, perform data preprocessing on the data to be labeled, and form preprocessed data to be labeled;

[0030] Preliminary annotation module: used to perform preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data;

[0031] Display module: used to render the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data;

[0032] Control and review module: used to perform quality control and review on the revised annotation data according to preset rules to obtain reviewed annotation data;

[0033] Extension and customization module: used to expand and customize the reviewed annotation data based on customized requirements to form customized annotation data.

[0034] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the processor runs a computer program or code stored in the memory to implement any of the semi-automatic labeling methods described above.

[0035] In addition, an embodiment of the present invention further provides a computer-readable storage medium for storing a computer program or code. When the computer program or code is executed by a processor, the semi-automatic labeling method as described in any one of the above is implemented.

[0036] In an embodiment of the present invention, a large amount of data can be quickly processed through initial screening, and content that obviously does not conform to the labeling rules can be filtered out or simple labeling tasks can be completed directly, which significantly reduces the need for manual intervention, thereby greatly improving the overall labeling speed; by reducing dependence on manual labelers through automation, the cost of labeling projects can be greatly reduced; even in the case of manual review, the overall cost is still much lower than full manual labeling due to the reduced workload; the automatic labeling engine provides relatively stable initial labeling, combined with manual review, which can further ensure the accuracy and consistency of the labeling results and reduce human errors; by collecting manually corrected data, the automatic labeling engine can be continuously learned and optimized. Over time, the accuracy and efficiency of automatic labeling will continue to improve, forming a virtuous circle. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1is a flow chart of a semi-automatic annotation method for improving annotation accuracy and efficiency in an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of the structure of a semi-automatic marking device for improving marking accuracy and efficiency in an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] For example, see Figure 1 , Figure 1 It is a flow chart of a semi-automatic annotation method for improving annotation accuracy and efficiency in an embodiment of the present invention.

[0043] like Figure 1 As shown, a semi-automatic annotation method for improving annotation accuracy and efficiency, the method comprising:

[0044] S11: obtaining data to be labeled, and performing data preprocessing on the data to be labeled to form preprocessed data to be labeled;

[0045] In the specific implementation process of the present invention, the data to be labeled is preprocessed to form preprocessed data to be labeled, including: performing format conversion processing on the data to be labeled, converting the data to be labeled from different sources into data to be labeled in a unified format; performing data cleaning and standardization processing on the data to be labeled in a unified format to obtain standardized data to be labeled; performing preliminary analysis on the standardized data to be labeled, and extracting feature data that is helpful for labeling based on the preliminary analysis results to form preprocessed data to be labeled.

[0046] Specifically, after obtaining the data to be labeled, in order to ensure that the format of the original data is unified and the quality meets the standards, and to prepare for subsequent labeling, corresponding format conversion, cleaning and standardization and feature extraction are required; that is, the data to be labeled is format converted, and the data to be labeled from different sources is converted into data to be labeled in a unified format; then the data to be labeled in a unified format is cleaned and standardized to obtain standardized data to be labeled; wherein cleaning and standardization are performed to remove invalid, repeated or erroneous data in the data to be labeled, denoise and scale the images in the data to be labeled, and segment and remove stop words from the text data in the data to be labeled; a preliminary analysis is performed on the standardized data to be labeled, and feature data that is helpful for labeling is extracted based on the preliminary analysis results to form preprocessed data to be labeled; a preliminary analysis is performed on the standardized data to be labeled to extract features that are helpful for subsequent labeling.

[0047] S12: performing preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data;

[0048] In the specific implementation process of the present invention, the automatic labeling engine performs preliminary labeling processing on the preprocessed data to be labeled to obtain preliminary labeled data, including: inputting the preprocessed data to be labeled into the automatic labeling engine for labeling processing according to the data type and label, and performing execution evaluation on the data labeling results to generate preliminary labeled data. The automatic labeling engine is based on historical data labeled according to data type and requirements, and then input into a deep learning model for training and generation.

[0049] Specifically, during the initial annotation, the pre-processed data to be annotated needs to be input into the automatic annotation engine for annotation processing according to the data type and annotation, and the data annotation results are evaluated for execution to generate preliminary annotated data; the automatic annotation engine here is annotated according to the data type and requirements based on historical data, and then input into the deep learning model (such as CNN, RNN, Transformer, etc.) for training and generation; in model selection and training, the appropriate pre-trained model can be selected according to the data type and annotation requirements, and fine-tuned on a specific data set; during the initial annotation, the automatic annotation engine automatically analyzes the data, labels the objects in the images in the annotated data, classifies the text, or transcribes the speech, etc.; after completing the preliminary annotation, the automatic annotation results are assigned a confidence score to provide a reference for subsequent manual review.

[0050] S13: Rendering the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data;

[0051] In the specific implementation process of the present invention, the preliminary annotation data is rendered into the human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data, including: rendering the preliminary annotation data according to the target object marked by the annotation box, point or line to obtain a rendering result; loading the rendering result into the human-computer interaction interface for display processing; the annotator performs manual annotation correction processing according to the preliminary annotation data displayed on the human-computer interaction interface to obtain corrected annotation data.

[0052] Furthermore, after obtaining the corrected annotation data, the method further includes: returning the corrected annotation data to the automatic annotation engine, and performing a correction process on the node parameters of the automatic annotation engine.

[0053] Specifically, in order to provide an efficient and intuitive interface, it is convenient for annotators to review and correct preliminary annotation data; the preliminary annotation data needs to be rendered to the human-computer interaction interface; when rendering, for intuitive display, the preliminary annotation data is rendered according to the target objects marked by the annotation boxes, points or lines to obtain the rendering results; then, the preliminary annotation data rendered to the interface is provided with batch modification, shortcut key operation, intelligent suggestions and other tools through the quick marking tool to reduce manual operation time; and the annotation progress is displayed through progress tracking and feedback, allowing annotators to provide feedback on the system to promote continuous optimization.

[0054] The intelligent learning and feedback mechanism continuously optimizes the automatic labeling engine by recording manual corrections. The manually corrected data is fed back to the automatic labeling engine. By iterating the automatic labeling engine, that is, adjusting the node parameters of the automatic labeling engine regularly or dynamically based on feedback, the accuracy and adaptability of automatic labeling are improved. Through active learning, uncertain or difficult cases are selected and pushed to manual labeling first, so as to guide model learning more effectively.

[0055] S14: performing labeling quality control and review processing on the corrected labeling data according to preset rules to obtain reviewed labeling data;

[0056] In the specific implementation process of the present invention, the revised annotation data is subjected to annotation quality control and review processing according to preset rules to obtain the reviewed annotation data, including: using an automatic annotation checking algorithm to check logical errors and inconsistencies in the revised annotation data, and obtaining the number of logical errors and inconsistencies in the revised annotation data; when the number of logical errors and inconsistencies in the revised annotation data is greater than a preset number, the revised annotation data enters a manual review process, and is subjected to expert verification processing in the manual review process to form the reviewed annotation data.

[0057] Specifically, in order to ensure the quality of the corrected annotation data and ensure the accuracy and consistency of the corrected annotation data, automatic inspection is required. The automatic annotation inspection algorithm is used to check the logical errors and inconsistencies in the corrected annotation data, and the number of logical errors and inconsistencies in the corrected annotation data is obtained; a preset number is set, and when the number of logical errors and inconsistencies in the corrected annotation data is less than or equal to the preset number, no manual review is required; when the number of logical errors and inconsistencies in the corrected annotation data is greater than the preset number, the corrected annotation data needs to enter the manual review process, and expert verification is performed in the manual review process to form the reviewed annotation data; for key data or difficult data cases, expert review is performed to ensure the annotation quality.

[0058] S15: Expand and customize the reviewed annotation data based on customized requirements to form customized annotation data.

[0059] In the specific implementation process of the present invention, the reviewed annotation data is expanded and customized based on the customized requirements to form customized annotation data, including: based on the data format and annotation rules in the customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

[0060] Specifically, in order to meet the demand for labeled data, by supporting customized requirements in different fields, such as data formats and labeling rules for specific industries, the reviewed labeled data is expanded and customized according to the data formats and labeling rules in the customized requirements to form customized labeled data.

[0061] In an embodiment of the present invention, a large amount of data can be quickly processed through initial screening, and content that obviously does not conform to the labeling rules can be filtered out or simple labeling tasks can be completed directly, which significantly reduces the need for manual intervention, thereby greatly improving the overall labeling speed; by reducing dependence on manual labelers through automation, the cost of labeling projects can be greatly reduced; even in the case of manual review, the overall cost is still much lower than full manual labeling due to the reduced workload; the automatic labeling engine provides relatively stable initial labeling, combined with manual review, which can further ensure the accuracy and consistency of the labeling results and reduce human errors; by collecting manually corrected data, the automatic labeling engine can be continuously learned and optimized. Over time, the accuracy and efficiency of automatic labeling will continue to improve, forming a virtuous circle.

[0062] For example 2, please refer to Figure 2 , Figure 2 It is a schematic diagram of the structural composition of a semi-automatic labeling device for improving labeling accuracy and efficiency in an embodiment of the present invention.

[0063] like Figure 2As shown, a semi-automatic annotation method for improving annotation accuracy and efficiency, the method comprising:

[0064] Data preprocessing module 21: used to obtain data to be labeled, and perform data preprocessing on the data to be labeled to form preprocessed data to be labeled;

[0065] In the specific implementation process of the present invention, the data to be labeled is preprocessed to form preprocessed data to be labeled, including: performing format conversion processing on the data to be labeled, converting the data to be labeled from different sources into data to be labeled in a unified format; performing data cleaning and standardization processing on the data to be labeled in a unified format to obtain standardized data to be labeled; performing preliminary analysis on the standardized data to be labeled, and extracting feature data that is helpful for labeling based on the preliminary analysis results to form preprocessed data to be labeled.

[0066] Specifically, after obtaining the data to be labeled, in order to ensure that the format of the original data is unified and the quality meets the standards, and to prepare for subsequent labeling, corresponding format conversion, cleaning and standardization and feature extraction are required; that is, the data to be labeled is format converted, and the data to be labeled from different sources is converted into data to be labeled in a unified format; then the data to be labeled in a unified format is cleaned and standardized to obtain standardized data to be labeled; wherein cleaning and standardization are performed to remove invalid, repeated or erroneous data in the data to be labeled, denoise and scale the images in the data to be labeled, and segment and remove stop words from the text data in the data to be labeled; a preliminary analysis is performed on the standardized data to be labeled, and feature data that is helpful for labeling is extracted based on the preliminary analysis results to form preprocessed data to be labeled; a preliminary analysis is performed on the standardized data to be labeled to extract features that are helpful for subsequent labeling.

[0067] The preliminary annotation module 22 is used to perform preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data;

[0068] In the specific implementation process of the present invention, the automatic labeling engine performs preliminary labeling processing on the preprocessed data to be labeled to obtain preliminary labeled data, including: inputting the preprocessed data to be labeled into the automatic labeling engine for labeling processing according to the data type and label, and performing execution evaluation on the data labeling results to generate preliminary labeled data. The automatic labeling engine is based on historical data labeled according to data type and requirements, and then input into a deep learning model for training and generation.

[0069] Specifically, during the initial annotation, the pre-processed data to be annotated needs to be input into the automatic annotation engine for annotation processing according to the data type and annotation, and the data annotation results are evaluated for execution to generate preliminary annotated data; the automatic annotation engine here is annotated according to the data type and requirements based on historical data, and then input into the deep learning model (such as CNN, RNN, Transformer, etc.) for training and generation; in model selection and training, the appropriate pre-trained model can be selected according to the data type and annotation requirements, and fine-tuned on a specific data set; during the initial annotation, the automatic annotation engine automatically analyzes the data, labels the objects in the images in the annotated data, classifies the text, or transcribes the speech, etc.; after completing the preliminary annotation, the automatic annotation results are assigned a confidence score to provide a reference for subsequent manual review.

[0070] Display module 23: used to render the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data;

[0071] In the specific implementation process of the present invention, the preliminary annotation data is rendered into the human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data, including: rendering the preliminary annotation data according to the target object marked by the annotation box, point or line to obtain a rendering result; loading the rendering result into the human-computer interaction interface for display processing; the annotator performs manual annotation correction processing according to the preliminary annotation data displayed on the human-computer interaction interface to obtain corrected annotation data.

[0072] Furthermore, after obtaining the corrected annotation data, the method further includes: returning the corrected annotation data to the automatic annotation engine, and performing a correction process on the node parameters of the automatic annotation engine.

[0073] Specifically, in order to provide an efficient and intuitive interface, it is convenient for annotators to review and correct preliminary annotation data; the preliminary annotation data needs to be rendered to the human-computer interaction interface; when rendering, for intuitive display, the preliminary annotation data is rendered according to the target objects marked by the annotation boxes, points or lines to obtain the rendering results; then, the preliminary annotation data rendered to the interface is provided with batch modification, shortcut key operation, intelligent suggestions and other tools through the quick marking tool to reduce manual operation time; and the annotation progress is displayed through progress tracking and feedback, allowing annotators to provide feedback on the system to promote continuous optimization.

[0074] The intelligent learning and feedback mechanism continuously optimizes the automatic labeling engine by recording manual corrections. The manually corrected data is fed back to the automatic labeling engine. By iterating the automatic labeling engine, that is, adjusting the node parameters of the automatic labeling engine regularly or dynamically based on feedback, the accuracy and adaptability of automatic labeling are improved. Through active learning, uncertain or difficult cases are selected and pushed to manual labeling first, so as to guide model learning more effectively.

[0075] Control and review module 24: used to perform quality control and review on the revised annotation data according to preset rules to obtain reviewed annotation data;

[0076] In the specific implementation process of the present invention, the revised annotation data is subjected to annotation quality control and review processing according to preset rules to obtain the reviewed annotation data, including: using an automatic annotation checking algorithm to check logical errors and inconsistencies in the revised annotation data, and obtaining the number of logical errors and inconsistencies in the revised annotation data; when the number of logical errors and inconsistencies in the revised annotation data is greater than a preset number, the revised annotation data enters a manual review process, and is subjected to expert verification processing in the manual review process to form the reviewed annotation data.

[0077] Specifically, in order to ensure the quality of the corrected annotation data and ensure the accuracy and consistency of the corrected annotation data, automatic inspection is required. The automatic annotation inspection algorithm is used to check the logical errors and inconsistencies in the corrected annotation data, and the number of logical errors and inconsistencies in the corrected annotation data is obtained; a preset number is set, and when the number of logical errors and inconsistencies in the corrected annotation data is less than or equal to the preset number, no manual review is required; when the number of logical errors and inconsistencies in the corrected annotation data is greater than the preset number, the corrected annotation data needs to enter the manual review process, and expert verification is performed in the manual review process to form the reviewed annotation data; for key data or difficult data cases, expert review is performed to ensure the annotation quality.

[0078] Extension and customization module 25: used to expand and customize the reviewed annotation data based on customized requirements to form customized annotation data.

[0079] In the specific implementation process of the present invention, the reviewed annotation data is expanded and customized based on the customized requirements to form customized annotation data, including: based on the data format and annotation rules in the customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

[0080] Specifically, in order to meet the demand for labeled data, by supporting customized requirements in different fields, such as data formats and labeling rules for specific industries, the reviewed labeled data is expanded and customized according to the data formats and labeling rules in the customized requirements to form customized labeled data.

[0081] In an embodiment of the present invention, a large amount of data can be quickly processed through initial screening, and content that obviously does not conform to the labeling rules can be filtered out or simple labeling tasks can be completed directly, which significantly reduces the need for manual intervention, thereby greatly improving the overall labeling speed; by reducing dependence on manual labelers through automation, the cost of labeling projects can be greatly reduced; even in the case of manual review, the overall cost is still much lower than full manual labeling due to the reduced workload; the automatic labeling engine provides relatively stable initial labeling, combined with manual review, which can further ensure the accuracy and consistency of the labeling results and reduce human errors; by collecting manually corrected data, the automatic labeling engine can be continuously learned and optimized. Over time, the accuracy and efficiency of automatic labeling will continue to improve, forming a virtuous circle.

[0082] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program on the computer-readable storage medium, and when the program is executed by a processor, the semi-automatic annotation method of any one of the above embodiments is implemented. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that stores or transmits information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.

[0083] An embodiment of the present invention further provides a computer application program, which runs on a computer and is used to execute the semi-automatic labeling method of any one of the above embodiments.

[0084] also, Figure 3 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention.

[0085] The embodiment of the present invention further provides an electronic device, such as Figure 3The electronic device includes a processor 302, a memory 303, an input unit 304, a display unit 305 and other devices. Those skilled in the art will understand that Figure 3 The structural components of the electronic device shown do not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 303 can be used to store the application 301 and various functional modules, and the processor 302 runs the application 301 stored in the memory 303, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both internal and external memories. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a U disk, a magnetic tape, etc. The memory disclosed in the present invention includes but is not limited to these types of memories. The memory disclosed in the present invention is only used as an example and not as a limitation.

[0086] The input unit 304 is used to receive the input of the signal and the keyword input by the user. The input unit 304 may include a touch panel and other input devices. The touch panel can collect the user's touch operation on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a playback control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 305 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 305 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 302 is the control center of the terminal device, which uses various interfaces and lines to connect the various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 303, and calling the data stored in the memory.

[0087] As an embodiment, the electronic device includes: one or more processors 302, a memory 303, and one or more applications 301, wherein the one or more applications 301 are stored in the memory 303 and are configured to be executed by the one or more processors 302, and the one or more applications 301 are configured to execute the semi-automatic labeling method in any one of the above-mentioned embodiments.

[0088] In an embodiment of the present invention, a large amount of data can be quickly processed through initial screening, and content that obviously does not conform to the labeling rules can be filtered out or simple labeling tasks can be completed directly, which significantly reduces the need for manual intervention, thereby greatly improving the overall labeling speed; by reducing dependence on manual labelers through automation, the cost of labeling projects can be greatly reduced; even in the case of manual review, the overall cost is still much lower than full manual labeling due to the reduced workload; the automatic labeling engine provides relatively stable initial labeling, combined with manual review, which can further ensure the accuracy and consistency of the labeling results and reduce human errors; by collecting manually corrected data, the automatic labeling engine can be continuously learned and optimized. Over time, the accuracy and efficiency of automatic labeling will continue to improve, forming a virtuous circle.

[0089] In addition, the above is a detailed introduction to a semi-automatic annotation method and related devices for improving annotation accuracy and efficiency provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A semi-automatic annotation method for improving annotation accuracy and efficiency, characterized in that: The method comprises: Obtaining data to be labeled, and performing data preprocessing on the data to be labeled to form preprocessed data to be labeled; Perform preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data; Rendering the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data; Performing labeling quality control and review processing on the revised labeling data according to preset rules to obtain reviewed labeling data; Based on customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

2. The semi-automatic labeling method according to claim 1, characterized in that: The step of preprocessing the data to be labeled to form preprocessed data to be labeled includes: Performing format conversion processing on the data to be annotated, converting the data to be annotated from different sources into data to be annotated in a unified format; Perform data cleaning and standardization on the data to be labeled in a unified format to obtain standardized data to be labeled; A preliminary analysis is performed on the standardized data to be labeled, and feature data helpful for labeling is extracted based on the preliminary analysis results to form preprocessed data to be labeled.

3. The semi-automatic labeling method according to claim 1, characterized in that: The method of performing preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data includes: The preprocessed data to be labeled is input into the automatic labeling engine for labeling according to the data type and label, and the data labeling result is evaluated for execution to generate preliminary labeled data. The automatic labeling engine is generated by inputting historical data into the deep learning model for training after labeling according to the data type and requirements.

4. The semi-automatic labeling method according to claim 1, characterized in that: The rendering of the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performing annotation correction processing based on the displayed preliminary annotation data to obtain the corrected annotation data, includes: Rendering the preliminary marked data according to the target object marked by the marked box, point or line to obtain a rendering result; Loading the rendering result into the human-computer interaction interface for display processing; The annotator performs manual annotation correction processing according to the preliminary annotation data displayed on the human-computer interaction interface to obtain corrected annotation data.

5. The semi-automatic labeling method according to claim 1, characterized in that: After obtaining the corrected annotation data, the method further includes: The corrected annotation data is returned to the automatic annotation engine, and the automatic annotation engine is subjected to correction processing of node parameters.

6. The semi-automatic labeling method according to claim 1, characterized in that: The step of performing labeling quality control and review processing on the revised labeling data according to preset rules to obtain the reviewed labeling data includes: Using an automatic annotation checking algorithm to check the logical errors and inconsistencies in the corrected annotation data, and obtaining the number of logical errors and inconsistencies in the corrected annotation data; When the number of logical errors and inconsistencies in the corrected annotation data is greater than a preset number, the corrected annotation data is put into a manual review process, and an expert verification process is performed in the manual review process to form reviewed annotation data.

7. The semi-automatic labeling method according to claim 1, characterized in that: The process of expanding and customizing the reviewed annotation data based on the customization requirements to form customized annotation data includes: Based on the data format and annotation rules in the customized requirements, the reviewed annotation data is expanded and customized to form customized annotation data.

8. A semi-automatic annotation method for improving annotation accuracy and efficiency, characterized in that: The method comprises: Data preprocessing module: used to obtain the data to be labeled, perform data preprocessing on the data to be labeled, and form preprocessed data to be labeled; Preliminary annotation module: used to perform preliminary annotation processing on the pre-processed data to be annotated based on the automatic annotation engine to obtain preliminary annotated data; Display module: used to render the preliminary annotation data into a human-computer interaction interface for display processing, and the annotator performs annotation correction processing based on the displayed preliminary annotation data to obtain corrected annotation data; Control and review module: used to perform quality control and review on the revised annotation data according to preset rules to obtain reviewed annotation data; Extension and customization module: used to expand and customize the reviewed annotation data based on customized requirements to form customized annotation data.

9. An electronic device, comprising a processor and a memory, characterized in that: The processor runs a computer program or code stored in the memory to implement the semi-automatic labeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing a computer program or code, characterized in that: When the computer program or code is executed by a processor, the semi-automatic labeling method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Label data auditing method and device, electronic equipment and storage medium

    CN116468951A

  • Automatic data labeling method and system

    CN116662881A

  • Automatic data labeling method and device based on multi-source power data

    CN118094234A

  • Device and method to speed up annotation quality check process

    EP4092565A1

Cited By

  • Blood vessel labeling verification method and device based on SYNTAX labeling specification, terminal and medium

    CN119742034A