Data cleaning method combined with data annotation
By combining data annotation and cleaning processes and using semantic information to guide the cleaning process, the problems of poor data quality and low processing efficiency are solved, achieving efficient and accurate data processing and meeting the needs of the fields of artificial intelligence and big data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, data labeling and data cleaning are usually done independently, resulting in poor data quality and low processing efficiency, which cannot meet the high-quality data requirements of the fields of artificial intelligence and big data.
By combining data annotation and data cleaning processes, semantic information is mined during the annotation process to guide the cleaning process, establishing mapping relationships, optimizing the annotation and cleaning process, and using deep learning algorithms to automatically generate preliminary annotation results and perform intelligent evaluation.
It improves the accuracy and efficiency of data annotation, ensures data quality, meets the demand for high-quality data, simplifies the data processing process, and enhances the standardization and consistency of data.
Smart Images
Figure CN121765091A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of data processing and data cleaning technologies, and particularly to a data cleaning method combining data annotation. Data annotation refers to adding labels, annotations, and other information to raw data (such as images, text, and audio) to imbue the data with semantic information, thereby facilitating computer understanding and processing. It is commonly used in machine learning and deep learning model training. Data cleaning, on the other hand, processes noisy data, duplicate data, missing values, and erroneous values in the raw data to improve its integrity, consistency, and accuracy; it is a crucial step in data preprocessing. Both have wide and critical applications in fields such as artificial intelligence, big data analysis, and intelligent recommendation. Background Technology
[0002] In current data processing workflows, data annotation and data cleaning are typically relatively independent steps. During data annotation, annotators may produce errors or non-standard results due to a lack of unified standards or misunderstandings. Simultaneously, the raw data itself may contain significant noise and errors; labeling it directly without cleaning increases workload and error rates. Traditional data cleaning methods often rely solely on statistical characteristics or simple rules, failing to fully utilize semantic information and struggle to accurately identify and process complex errors. Furthermore, the separate data annotation and cleaning processes make data processing cumbersome, inefficient, and difficult to guarantee data quality, failing to meet the urgent demand for high-quality data in fields such as artificial intelligence and big data. Summary of the Invention
[0003] To address the aforementioned technical issues, this invention provides a data cleaning method that combines data annotation with data cleaning. This method organically integrates the data annotation and data cleaning processes, mining semantic information from the data through the annotation process, and using this semantic information to guide data cleaning, thereby improving the accuracy and relevance of data cleaning. Simultaneously, the annotation process is optimized based on the cleaned data, improving annotation quality and efficiency. This provides high-quality, standardized data for subsequent data analysis and model training, solving the problems of poor data quality and low processing efficiency caused by the separation of data annotation and cleaning in existing data processing workflows.
[0004] The technical solution of this invention is: A data cleaning method incorporating data labeling includes the following steps: Text and image annotation: A highly integrated annotation function system is constructed. For text data, a visual operation interface is used to select text fragments and add keywords, entity names, and sentiment category tags using intelligent prompt drop-down menus. At the same time, the intelligent semantic analysis module analyzes the semantics of the text based on the built-in knowledge graph and recommends tag options. For image data, rectangular boxes, polygonal boxes, and point annotation methods are provided. An automatic snapping function is used to capture the edge feature points of irregular objects for annotation. Undo, redo, copy and paste operations are supported during the annotation process, as well as annotation transparency adjustment and annotation box color customization settings. An annotation model is built based on deep learning algorithms and existing annotation data to automatically generate preliminary annotation results for new data.
[0005] Based on the annotation results, data cleaning is performed: a mapping relationship is established between the annotation results and data cleaning rules. For text data, the annotation information is analyzed. When contradictory sentiment tags or annotation information that is seriously inconsistent with the text content are found, the data is judged as erroneous and deleted. At the same time, the consistency of the annotations of the same entity in the text is checked. For image data, the labeled target objects are compared with the actual content using image recognition algorithms. When there are serious deviations in category, position, or shape, or problems with image integrity and clarity, the data is judged as erroneous and deleted. At the same time, the consistency of the annotations of similar target objects in the image is checked. An intelligent evaluation strategy is adopted to assess the impact of missing annotations on the data value. If the impact is lower than the predetermined value, it is filled with default values or supplemented by interpolation using similar data. If the impact is higher than the predetermined value, it is deleted. Based on the annotation and cleaning results, data standardization processing is performed: for text data, the annotated text data is converted according to a unified encoding format and character set, and the text length is normalized by adding filler words, truncation, or summary extraction; for image data, the images are adjusted to the specified size, resolution, and color mode, and the coordinate information of the annotations is converted according to a unified coordinate system; the annotation labels are standardized, labels with similar meanings are merged, and the naming and encoding methods of the labels are unified. Furthermore, The intelligent semantic analysis module performs semantic parsing on selected text fragments based on natural language processing algorithms and knowledge graphs. Furthermore, The annotation model is constructed based on convolutional neural networks or recurrent neural networks. Furthermore, The image data is checked for integrity and clarity, and an image quality evaluation algorithm is used to determine whether the image has serious blurring or missing parts. Furthermore, In text data, if adjectives are missing, adjectives with a frequency higher than a predetermined value are extracted from similar texts to fill in the missing ones; in image data, if some attribute labels of the target object are missing, they are filled in by interpolation based on the average attribute values of similar target objects.
[0006] Furthermore, When normalizing text length, the rules for adding filler words are set based on the text's theme and grammatical structure. Furthermore, Based on different application scenarios and algorithm requirements, corresponding image size standards are set. For image classification tasks, the pixels of the images are uniformly adjusted; for object detection tasks, the image size is adjusted to ensure the integrity and clarity of the target object.
[0007] In terms of resolution adjustment, a high-quality image interpolation algorithm is used to ensure that the image does not exhibit blurring, jagged edges, or other distortions during scaling; at the same time, the color mode of the image is uniformly converted to RGB mode.
[0008] When adjusting the resolution of the image data, a bicubic interpolation algorithm can be used.
[0009] The beneficial effects of this invention are Highly efficient and accurate annotation capabilities: The highly integrated annotation system constructed by this method greatly improves the efficiency and accuracy of text and image data annotation. For text annotation, the visual interface, combined with intelligent suggestion drop-down menus, intelligent semantic analysis modules, and knowledge graphs, can quickly locate key information and recommend tags, reducing the time spent on manual retrieval and judgment by annotators. For image annotation, rich annotation tools and automatic snapping functions cater to the annotation needs of objects with different shapes, and also support various convenient operations and personalized settings, flexibly handling complex annotation scenarios. Simultaneously, the automatic annotation assistance function, based on deep learning algorithms, can automatically generate preliminary annotation results. Annotators only need to review and correct them, significantly reducing the workload of manual annotation. Furthermore, the automatic annotation result evaluation function continuously optimizes the model, constantly improving annotation efficiency and accuracy, and significantly shortening the data annotation cycle. A comprehensive and intelligent data cleaning mechanism: A rigorous mapping mechanism between annotation results and data cleaning rules enables deep cleaning of text and image data. Through multi-dimensional analysis of annotation information, such as sentiment contradiction detection and semantic logic verification in text, and target comparison, integrity, and clarity checks in images, erroneous data can be accurately identified and promptly deleted. For missing annotation data, the intelligent evaluation strategy can flexibly handle the situation based on the degree of impact; data with minor impact is supplemented through filling or interpolation, while data with major impact is discarded, ensuring data quality while maximizing the preservation of data value. The consistency check function ensures the uniformity of annotations for the same entity or similar targets, effectively avoiding data errors caused by annotation chaos and laying a solid data foundation for subsequent data analysis. Standardized data processing: Standardized rules for text and image data significantly enhance data compatibility and standardization. A unified encoding format and character set for text data resolves cross-platform data transmission issues; multiple normalization methods eliminate text length differences, facilitating subsequent analysis and processing. Standardized conversions of image data in terms of size, resolution, color mode, and coordinate information cater to different application scenarios and algorithm requirements, ensuring consistent data performance across various image tasks. Standardized label processing merges similar labels, unifies naming and encoding, eliminates semantic ambiguity, and achieves high uniformity in data format and semantics, improving data universality and understandability, facilitating data storage, analysis, and application, and reducing data processing costs and complexity. Attached Figure Description
[0010] Figure 1 This is a schematic diagram of the workflow of the present invention. Detailed Implementation
[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0012] This invention discloses a data cleaning method combined with data annotation, comprising the following steps: 1. Provide text and image annotation functions: This invention constructs a highly integrated and flexible annotation function system, providing comprehensive support for text and image data annotation. For text data annotation, diverse and convenient annotation tools have been carefully designed. After entering the visual operation interface, annotators can accurately select text fragments with the mouse and quickly add tags such as keywords, entity names, and sentiment categories using the intelligent suggestion drop-down menu. To further improve the accuracy and efficiency of annotation, an intelligent semantic analysis module is integrated. This module can perform semantic analysis on the selected text fragments in real time and recommend possible tag options to annotators based on the built-in knowledge graph. For example, when an annotator selects a text describing "XX Company released a new mobile phone," it automatically identifies "XX Company" as an entity name and recommends relevant tags such as "enterprise" and "technology company," assisting the annotator in quickly completing the annotation work. For image data annotation, it offers a rich and practical range of annotation methods, including rectangular boxes, polygonal boxes, and point annotations. Annotators can flexibly choose the appropriate annotation tool based on the shape and characteristics of the target object in the image. When using polygonal boxes to annotate irregular objects, it features an automatic snapping function that automatically captures contour feature points when the annotation point is close to the object's edge, significantly improving the accuracy and efficiency of annotation. It also supports common operations such as undo, redo, and copy-paste during the annotation process, facilitating the modification and adjustment of annotation results. Furthermore, to meet the annotation needs of different scenarios, it also includes personalized functions such as adjusting annotation transparency and customizing annotation box colors, making the annotation process more user-friendly.
[0013] To effectively improve annotation efficiency, an advanced automatic annotation assistance function has been introduced. This function, based on deep learning algorithms and a large amount of existing annotation data, constructs a powerful annotation model. When new data enters the system, the annotation model automatically analyzes and predicts, generating preliminary annotation results. For example, in text sentiment annotation tasks, the model determines the sentiment tendency of the text based on its vocabulary, grammatical structure, and semantic information, and assigns it corresponding sentiment tags. Annotators only need to review and correct the preliminary annotation results, greatly reducing the workload of manual annotation. Simultaneously, it also features an automatic annotation result evaluation function, which can optimize and adjust the annotation model in real time based on the annotation personnel's corrections, continuously improving the accuracy of automatic annotation.
[0014] 2. Based on the annotation results, perform data cleaning: In the data cleaning stage, a rigorous and efficient mapping mechanism between annotation results and data cleaning rules was established. When dealing with text data, the annotation information undergoes in-depth analysis. If multiple contradictory sentiment labels appear, such as a text fragment being labeled as both "positive" and "negative," a data error judgment mechanism is triggered, marking the data as erroneous and deleting it. Furthermore, the semantic logic of the text is checked; if a serious discrepancy is found between the annotation information and the text content, it is also judged as erroneous data. For image data, image recognition algorithms are used to compare and analyze the labeled target objects with the actual content. If the labeled target object deviates significantly from the actual object in the image in terms of category, location, or shape (e.g., labeling a cat as a dog), or if the labeled box differs too much from the actual object's boundary, the data will be judged as erroneous and deleted. Simultaneously, the integrity and clarity of the image are also checked; images with severe blurring or missing parts will also be included in the erroneous data processing. For data with missing annotations, an intelligent evaluation strategy is adopted. First, the impact degree of the missing part on the overall data value is evaluated. If the missing part has little impact on the core value of the data, for example, a small number of insignificant keywords are missing in text data, or the annotation of the background area that does not affect the recognition of the main target object is missing in image data, according to the preset rules, it is processed by filling default values or interpolating with similar data. Specifically, in text data, if an adjective is missing, the adjectives with higher frequencies of occurrence are extracted from similar texts for filling; in image data, if some attribute annotations of the target object are missing, interpolation is performed according to the average attribute values of the same type of target objects. If the missing part has a great impact on the data value, such as the key entity name is missing in text data, or the main target object annotation is missing in image data, then the data will be果断 deleted.
[0015] 3. According to the annotation and cleaning results, data standardization processing is carried out: In the data standardization processing stage, a series of detailed and strict standardization rules are formulated according to the characteristics of text and image data. For text data, first, the annotated text data is converted according to a unified encoding format (such as UTF-8) and character set to ensure the compatibility of the data between different systems and platforms. At the same time, to eliminate the impact of text length differences on subsequent data analysis, a variety of normalization methods are adopted. For example, for shorter texts, the length is expanded to a specified length by adding filler words (such as function words like "的" and "了"); for longer texts, truncation or summary extraction methods are used to retain key information and compress the text length within a specified range. In the processing of image data, the images are uniformly adjusted to the specified size, resolution, and color mode. Specifically, according to different application scenarios and algorithm requirements, corresponding image size standards are set. For example, for image classification tasks, the images are uniformly adjusted to 224×224 pixels; for object detection tasks, the images are adjusted to an appropriate size to ensure the integrity and clarity of the target object. In terms of resolution adjustment, a high-quality image interpolation algorithm (such as the bicubic interpolation algorithm) is adopted to ensure that there are no distortion phenomena such as blurring and jagging during the scaling of the images. At the same time, the color mode of the images is uniformly converted to the RGB mode to ensure data consistency. In addition, for the annotated coordinate information, it is converted according to a unified coordinate system to ensure the consistency of the coordinate representation of image data from different sources. The above is only the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.
Claims
1. A data cleaning method combined with data labeling, characterized in that, comprising the following steps: labeling text and pictures: constructing a highly integrated labeling function system, for text data, through a visual operation interface, using an intelligent prompt drop-down menu to select text segments to add keywords, entity names, and sentiment category labels, while using an intelligent semantic analysis module to analyze text semantics based on an embedded knowledge graph and recommend label options; for picture data, providing rectangle, polygon, and point labeling methods, using automatic adsorption function to capture irregular object edge feature points for labeling, and supporting undo, redo, copy and paste operations during labeling, as well as labeling transparency adjustment and labeling box color customization; based on deep learning algorithms and existing labeled data, a labeling model is constructed to automatically generate preliminary labeling results for new data; According to the labeling results, data cleaning is performed: a mapping relationship between labeling results and data cleaning rules is established, for text data, analyze the labeling information, when there are contradictory sentiment labels or the labeling information is seriously inconsistent with the text content, determine it as error data and delete it, at the same time, check the consistency of the same entity labeling in the text; for picture data, compare the labeled target objects with the actual content through image recognition algorithm, when there are serious deviations in category, location or shape, and problems in picture integrity and clarity, determine it as error data and delete it, at the same time, check the consistency of the same target object labeling in the picture; using intelligent evaluation strategy, evaluate the influence of missing labeling part on data value, if the influence is lower than the predetermined value, fill in the default value or use similar data interpolation to supplement, if the influence is higher than the predetermined value, delete it; According to the labeling and cleaning results, data standardization processing is performed: for text data, the labeled text data is converted according to unified encoding format and character set, and the text length is normalized by adding filler words, truncation or abstract extraction; for picture data, the picture is adjusted to a specified size, resolution and color mode, and the coordinate information of the label is converted according to a unified coordinate system; standardize the labeling labels, merge labels with similar meanings, and unify the naming and coding method of the labels.
2. The method of claim 1, characterized in that, The intelligent semantic analysis module analyzes the semantics of the selected text segment based on natural language processing algorithms and knowledge graphs.
3. The method of claim 1, characterized in that, The labeling model is constructed based on convolutional neural networks or recurrent neural networks.
4. The method of claim 1, characterized in that, The completeness and clarity of the picture data is checked by using image quality evaluation algorithm to determine whether the picture has serious blur or missing problems.
5. The method of claim 1, characterized in that, In the text data, if the adjective is missing, extract adjectives with a frequency higher than a predetermined value from similar text to fill in; in the picture data, if part of the attribute labeling of the target object is missing, interpolate the average attribute value of the similar target object. 6. The method of claim 1, wherein, When normalizing the length of the text, the rule of adding fillers is set according to the text theme and the grammatical structure.
7. The method of claim 1, wherein, According to different application scenarios and algorithm requirements, set the corresponding picture size standard, for image classification tasks, adjust the pixels of the picture uniformly; for target detection tasks, adjust the size of the picture to ensure the integrity and clarity of the target object; In terms of resolution adjustment, high-quality image interpolation algorithms are used to ensure that the picture will not appear blurred, jagged and other distortion phenomena during scaling; at the same time, the color mode of the picture is uniformly converted to the RGB mode.
8. The method of claim 1 or 7, wherein, When adjusting the resolution of the picture data, a bicubic interpolation algorithm is used.