Intelligent data cleaning method, computer readable storage medium and terminal

Abstract

The application provides an intelligent data cleaning method, a computer readable storage medium and a terminal, and belongs to the fields of artificial intelligence and quality inspection.The intelligent data cleaning method comprises single-task data iteration based on model training and reasoning, single-task data iteration based on data analysis, and project data collection into a standard database, so as to expand the scale of the standard data set.The scheme promotes the continuous optimization iteration of data and models, forms a system solution scheme which is standard and objective, has efficient labeling, is convenient to query and comprehensive in analysis, and is convenient for popularization and application in the fields of quality inspection or detection.

Description

Technical Field

[0001] This invention belongs to the technical fields of artificial intelligence and industrial quality inspection, specifically relating to an intelligent data cleaning method, a computer-readable storage medium, and a terminal. Background Technology

[0002] High-quality data cleaning solutions are the foundation for building efficient and reliable intelligent industrial quality inspection systems. The core idea is to continuously improve the automation level of data annotation and expand the diversity of data while ensuring high-quality requirements such as unified and accurate data annotation standards. Currently, the main problems with this technology are: (1) how to build efficient and reliable quality inspection datasets with limited resources; (2) annotation consistency issues; and (3) the problem of obtaining rare defect samples.

[0003] Currently, the industrial quality inspection field lacks unified data standards and processing procedures. Much work focuses on developing standards and specifications, automated annotation tools, defect generation, and small-sample model solutions. Therefore, there is an urgent need for a systematic solution that is objective in its standards, efficient in its annotation, convenient in its querying, and comprehensive in its analysis. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide an intelligent data cleaning method, a computer-readable storage medium, and a terminal that can solve the above-mentioned problems.

[0005] Design Principles: Based on a multimodal industrial quality inspection model, a complete data processing and iteration process is established, aiming to create a system solution that is standardized, objective, efficiently labeled, conveniently queried, and comprehensively analytical. The method continuously iterates through data processing flows such as single-task data iteration based on model training and inference, single-task data iteration based on data analysis, and merging project data into a standard database. This systematically addresses the aforementioned issues, thereby promoting continuous optimization and iteration of data and models. The main processes include standard setting and alignment, automatic labeling, sample retrieval and filtering, and data expansion. This invention connects the above processing flows in the form of an industrial quality inspection intelligent agent. Each node represents a processing flow, and nodes communicate with each other using the Model Context Protocol (MCP). The overall scheme is as follows.

[0006] An intelligent data cleaning method includes: a first stage, single-task data iteration based on model training and inference, using the model to discover deficiencies in the data and expand the effective samples during project production; a second stage, single-task data iteration based on data analysis, performing data analysis from two aspects: data annotation quality and sample distribution, and guiding the data cleaning iteration; and a third stage, merging the project dataset into a standard database to expand the scale of the standard dataset.

[0007] Furthermore, the processing flow for the first-stage single-task data iteration includes labeling strategy design, model training, and active learning to mine uncertain samples.

[0008] The annotation strategy design includes automatic annotation of simple samples, batch annotation of complex samples, and batch rule processing of samples. The annotation includes image classification annotation and image segmentation annotation. Among them, automatic annotation of simple samples is achieved by using an automatic annotation model to automatically annotate simple samples; interactive and fast batch annotation of complex samples is performed to improve the interactive efficiency of annotation; and batch rule processing of samples is performed according to combination rules to make the samples meet certain rule constraints.

[0009] Image classification and annotation includes: first, extracting image features using a large industrial quality inspection model; second, using DBSCAN to cluster all images based on feature similarity, and manually labeling the overall categories of some well-consistent categories; third, automatically assigning categories to unlabeled data above a set confidence level, and manually confirming the remaining data; finally, performing cluster analysis based on clustering features, and manually adding, deleting, and modifying some category data; and optimizing the edge segmentation and alignment of special targets.

[0010] Clustering features include, but are not limited to, area, gradient, and shape. Special targets include black dots, white dots, etc., which are then segmented and optimized using thresholding.

[0011] Image segmentation and annotation includes: First, after annotating some data, inference and prediction are performed on the remaining data, and a low threshold (e.g., 0.2–0.3) is selected for prediction transformation to ensure that the target is predicted; Second, memory learning is performed on the existing segmentation data, and similar image targets are predicted through inference, and batch automatic annotation is performed by setting a high confidence threshold (e.g., similarity greater than 0.95); Third, the annotation is optimized using the SAM segmentation model, and when the IOU with the predicted annotation is greater than a certain threshold (recommended 0.9–0.95), the result of SAM segmentation is trusted and converted into an annotation; Finally, the annotation of the remaining data is confirmed or modified.

[0012] The model training uses an integrated algorithm development framework for training image classification, object detection, or semantic segmentation models, and provides a one-click deployment function for typical hardware (such as NVIDIA GPU graphics cards).

[0013] The sources of uncertain samples include: samples with low class confidence, samples with poor consistency between segmentation edges and SAM segmentation edges, and boundary samples of the data distribution. The first two are obtained through model inference, such as selecting samples with the lowest predicted output probability values ​​and samples with the smallest overlap with SAM segmentation. The third is obtained by sampling the farthest distance based on target features: in image classification, image features extracted from the industrial quality inspection model are used as target features; in image segmentation, target feature extraction first involves cropping an image of a specified size centered on the current target and masking other targets, then inputting it into the industrial quality inspection model to obtain the target features.

[0014] Furthermore, the second-stage single-task data iteration process includes image target library construction, image target feature extraction, clustering and distribution visualization analysis, sparse sample screening, fuzzy and isolated sample screening, and data augmentation generation.

[0015] The construction of the image target library includes: collecting all classification and segmentation data; for image classification data, taking all original images; and for image segmentation data, cropping a specified size region centered on the segmentation target as the image target.

[0016] Feature extraction of image targets includes: inputting images from the image target library into the industrial quality inspection large model, and the output is the image target features.

[0017] Clustering and distribution visualization analysis includes: the target feature clustering method is to calculate the Mahalanobis distance between all target features, cluster targets with a Mahalanobis distance less than 0.01, and manually confirm their categories based on typical examples of the clustering results; the distribution visualization analysis method is to use t-SNE to visualize the feature data in 2D or 3D.

[0018] Sparse sample filtering includes defining sample density as the number of targets within a specified Mahalanobis distance (e.g., setting the range to 0.1), and filtering out samples with a sample density less than a preset density threshold. Specifically, this involves filtering out samples with a sample density less than a certain threshold (e.g., setting the threshold to 3).

[0019] Fuzzy and isolated sample filtering involves: representing fuzzy samples as those where the inter-class distance is less than the intra-class distance, and isolated samples as those where only a few samples of the target exist within a unit Mahalanobis distance. Generally, the number of targets is less than 3.

[0020] Data augmentation generation includes: first, applying textures to sparse targets by rotating, scaling, adjusting brightness / contrast, slightly blurring, and adding noise, and then manually confirming the samples to be augmented; next, performing feature comparison in a standard target image library, searching for missing samples in the current data's potential spatial distribution from the standard target image library, generating transfer images, and manually confirming the samples to be augmented.

[0021] Furthermore, the third-stage standard dataset scaling process includes material category division and data category mapping, category de-obfuscation, data deduplication, data balanced sampling, and data update into the database.

[0022] Material category classification and data mapping include: first, finding 1 to 3 recommended classes in the standard library for each category in the project through feature similarity calculation, and then manually confirming the categories and constructing the mapping.

[0023] Category de-obfuscation processing includes: projecting the target features of the project data onto a standard data feature library to compare intra-class and inter-class similarities, and calculating the proportion of inter-class similarity greater than intra-class similarity.

[0024] Data deduplication involves removing duplicates from targets whose features have a cosine similarity greater than a certain threshold. This threshold can be set to 0.97.

[0025] Data equalization sampling includes: using the furthest distance algorithm to target and sample classes with an excessive number of classes.

[0026] The update process includes new category updates and data updates. When expanding to new categories, labels are incremented to avoid modifying previous data labels. Data updates directly incorporate project data into the standard dataset.

[0027] The present invention also provides a computer-readable storage medium storing computer instructions thereon, which execute the aforementioned intelligent data cleaning method when the computer instructions are run.

[0028] The present invention also provides a terminal, including a memory and a processor, wherein the memory stores a single-task dataset, a standard dataset, an image dataset, and computer instructions that can be executed on the processor, and the processor executes the aforementioned intelligent data cleaning method when executing the computer instructions.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: The solution of this application provides a whole-process processing method including model training and inference iteration, data analysis iteration, and standard dataset update. It focuses on mining "high-value difficult examples" and combining them with "balanced and clean structured data" to form the final "standard dataset". The solution promotes the continuous optimization and iteration of data and models, forming a system solution that is standard and objective, efficient in annotation, convenient in querying, and comprehensive in analysis, which is convenient for promotion and application in the fields of quality inspection or testing. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the intelligent data cleaning method of the present invention; Figure 2 The annotation process for image classification; Figure 3 The annotation process for image segmentation; Figure 4 A visual diagram illustrating the distribution of defective targets; Figure 5 This is a diagram illustrating interactive, rapid batch annotation. Figure 6 A method for classifying and mapping the target material of defects. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0033] An intelligent data cleaning method, such as Figure 1 As shown, it includes the following three stages.

[0034] In the first phase, single-task data iteration based on model training and inference utilizes the model to discover deficiencies in the data and expand the effective samples during the project production process.

[0035] In the second stage, single-task data iteration based on data analysis conducts data analysis from two aspects: data annotation quality and sample distribution, and guides data cleaning iteration.

[0036] The third phase involves expanding the size of the standard dataset by merging project data into a standard database.

[0037] The first phase mainly includes labeling strategy design, model training and inference, and active learning to mine uncertain samples. The specific implementation methods are as follows.

[0038] ① Labeling Strategy Implementation Plan: An automatic labeling model is used to achieve high-quality automatic labeling of simple samples and interactive, rapid batch labeling of complex samples, thereby improving the interactive efficiency of labeling. Finally, batch rule processing is performed on samples according to combination rules to ensure that the samples meet certain rule constraints. Industrial quality inspection data labeling mainly involves image classification labeling and image segmentation labeling.

[0039] ②The steps for image classification annotation are as follows: Figure 2 As shown, the specific steps include: (a) extracting image features using a large industrial quality inspection model; (b) using DBSCAN to cluster all images based on feature similarity, and manually labeling the overall categories of some well-consistent categories; (c) automatically assigning categories to unlabeled data with a certain confidence level or higher, and manually quickly confirming the rest; (d) performing cluster analysis based on features such as area, gradient, and shape, and manually adding, deleting, and modifying some category data; and optimizing the edge segmentation of some targets, such as threshold segmentation optimization for black and white dots.

[0040] ③ The annotation implementation steps for image segmentation are as follows: Figure 3 As shown, the specific steps include: (a) After labeling a portion of the data, performing inference and prediction on the remaining data, selecting a low threshold (e.g., 0.2–0.3) for prediction transformation to ensure that the vast majority of targets can be predicted. (b) Performing memory learning on the existing segmentation data, and predicting similar image targets through inference, and automatically labeling them with high confidence. (c) Using the SAM segmentation model for label optimization, when the IOU with the predicted label is greater than a certain threshold (recommended 0.9–0.95), choosing to trust the SAM segmentation result and converting it into a label; (d) Confirming or modifying the labels on the remaining data.

[0041] ④ The model training and inference process adopts an integrated algorithm development framework for training image classification, object detection, or semantic segmentation models, and provides a one-click deployment function for typical hardware (such as NVIDIA GPU graphics cards).

[0042] ⑤ Uncertain sample mining mainly originates from samples with low class confidence, samples with poor consistency between segmentation edges and SAM segmentation edges, and boundary samples of the data distribution. The first two can be obtained through model inference, such as selecting several samples with the lowest predicted output probability values ​​or several samples with the smallest overlap with SAM segmentation; the third is obtained by sampling the furthest distance based on the target features. In image classification, image features extracted by the industrial quality inspection large-scale model are used as target features; in image segmentation, target feature extraction first involves cropping an image of a specified size centered on the current target and masking other targets, then inputting it into the industrial quality inspection large-scale model to obtain the target features.

[0043] The second stage mainly includes image target library construction, image target feature extraction, clustering and distribution visualization analysis, sparse sample screening, fuzzy and isolated sample screening, and data augmentation generation. The specific implementation methods are as follows.

[0044] ① The image target library is constructed by collecting all classification and segmentation data; for image classification data, all original images are taken; for image segmentation data, a specified size region centered on the segmentation target is cropped, which is the image target.

[0045] ② The image target feature extraction method inputs images from the image target library into the industrial quality inspection large model, and the output is the image target features.

[0046] ③ The target feature clustering analysis method calculates the Mahalanobis distance between all target features, clusters targets with a Mahalanobis distance less than 0.01, and manually confirms their categories based on typical examples of the clustering results.

[0047] ④ Data distribution visualization analysis methods use t-SNE to visualize feature data in 2D or 3D. Figure 4 This is a visual distribution diagram of the defective samples.

[0048] ⑤ The sparse sample screening method defines the sample density as the number of targets within a specified Mahalanobis distance (e.g., setting the range to 0.1). The sparse sample screening method selects samples whose sample density is less than a certain threshold (e.g., setting the threshold to 3).

[0049] ⑥ The fuzzy and isolated sample screening method represents fuzzy samples as samples where the inter-class distance is less than the intra-class distance, while isolated samples represent samples where only a few sample targets exist within a unit Mahalanobis distance, generally with less than 3 targets.

[0050] ⑦ Data Augmentation Generation Method: First, for the aforementioned sparse targets, texture mapping is performed using transformations such as rotation, scaling, brightness / contrast adjustment, slight blurring, and noise addition. The required augmentation samples are then manually confirmed through interactive verification (see [link to relevant documentation]). Figure 5Then, feature comparison is performed in the standard target image library, and samples missing in the current data potential spatial distribution are searched from the standard target image library, transferred and generated, and the samples to be expanded are manually confirmed.

[0051] The third stage mainly includes material category classification and data category mapping, category de-obfuscation, data deduplication, data balanced sampling, and updating into the database, as detailed below.

[0052] ① Material category classification and mapping methods, such as Figure 6 As shown, firstly, 1 to 3 recommended classes in the standard library are found for each category in the project through feature similarity calculation, and then the category is manually confirmed and the mapping is constructed.

[0053] ② The category decongestion method projects the target features of the project data onto a standard data feature library to compare intra-class and inter-class similarities, and calculates the proportion of inter-class similarity that is greater than intra-class similarity.

[0054] ③ The data deduplication method removes duplicates from targets whose feature cosine similarity is greater than a certain threshold, which can be set to 0.97.

[0055] ④ The balanced sampling method uses the farthest distance algorithm to target and sample classes with an excessive number of classes.

[0056] ⑤ The update and entry methods mainly involve new category updates and data updates. When expanding new categories, the labels are incremented to avoid modifying the previous data labels. Data updates directly merge the project data into the standard dataset.

[0057] In summary, this application systematically analyzes and processes each stage of data cleaning, including three closed-loop iterative processes: single-task data iteration based on model training and inference, single-task data iteration based on data analysis, and merging project data into a standard database.

[0058] Single-task data iteration based on model training and inference focuses on using the model to discover deficiencies in the data and expand the effective samples during project production. The main processing flow includes manual annotation, model training, and active learning to mine uncertain samples. The model predicts unlabeled data and filters out samples that the model is "unsure" about. For the filtered samples, the current model is used to pre-label them, generating pseudo-labels. After manual correction or confirmation, the model is trained again, forming a closed loop of continuous optimization and iteration.

[0059] Single-task data iteration based on data analysis focuses on analyzing data annotation quality and sample distribution, guiding dataset cleaning iterations. The main processing flow includes image target library construction, image target feature extraction, clustering and distribution visualization analysis, sparse sample screening, fuzzy and isolated sample screening, and data augmentation and generation. By performing clustering and distribution visualization analysis on the samples in the current dataset, sparse, fuzzy, or isolated samples (usually mislabeled or extremely difficult examples) are screened. Data augmentation is then performed on these samples, or data is supplemented using generative models.

[0060] During the process of merging project data into the standard database, the data is first categorized by material type to ensure consistent label names across different batches. Secondly, a specific category de-obfuscation method is used to address the issue of overly similar features between two categories. Then, highly similar redundant samples are removed using a data deduplication method. Finally, data equalization sampling is used to resolve the class imbalance problem.

[0061] The intelligent data cleaning method proposed in this invention provides a complete process including model training and inference iteration, data analysis iteration, and standard dataset update. It focuses on mining "high-value difficult examples" and combining them with "balanced and clean structured data" to form the final "standard dataset".

[0062] This invention also provides a computer-readable storage medium storing computer instructions, which, when executed, perform the aforementioned intelligent data cleaning method. For detailed implementation methods, please refer to the foregoing description; they will not be repeated here.

[0063] This invention also provides a terminal, including a memory and a processor. The memory stores a single-task dataset, a standard dataset, an image dataset, and computer instructions executable on the processor. When the processor executes the computer instructions, it performs the aforementioned intelligent data cleaning method. For detailed implementation methods, please refer to the foregoing description; they will not be repeated here.

[0064] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including permanent and non-permanent, removable and non-removable media. Information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent data cleaning method, characterized in that, Intelligent data cleaning methods include: The first phase involves single-task data iteration based on model training and inference, using the model to discover deficiencies in the data and expand the effective samples during the project production process. The second stage involves single-task data iteration based on data analysis, which analyzes data from two aspects: data annotation quality and sample distribution, and guides data cleaning iteration. In the third stage, the project dataset is merged into the standard database to expand the size of the standard dataset.

2. The intelligent data cleaning method according to claim 1, characterized in that: The first phase of the single-task data iteration process includes labeling strategy design, model training, and active learning to mine uncertain samples.

3. The intelligent data cleaning method according to claim 2, characterized in that: The annotation strategy design includes automatic annotation of simple samples, batch annotation of complex samples, and batch rule processing of samples. The annotation includes image classification annotation and image segmentation annotation. Among them, automatic annotation of simple samples is achieved by using an automatic annotation model to automatically annotate simple samples; interactive and fast batch annotation of complex samples is performed to improve the interactive efficiency of annotation; and batch rule processing of samples is performed according to combination rules to make the samples meet certain rule constraints.

4. The intelligent data cleaning method according to claim 2, characterized in that, The sources of uncertain samples include: samples with low class confidence, samples with poor consistency between the segmentation edge and the SAM segmentation edge, and boundary samples of the data distribution.

5. The intelligent data cleaning method according to claim 1, characterized in that: The second-stage single-task data iteration process includes image target library construction, image target feature extraction, clustering and distribution visualization analysis, sparse sample screening, fuzzy and isolated sample screening, and data augmentation generation.

6. The intelligent data cleaning method according to claim 5, characterized in that, The construction of the image target library includes: collecting all classification and segmentation data; for image classification data, taking all original images; and for image segmentation data, cropping a specified size region centered on the segmentation target as the image target.

7. The intelligent data cleaning method according to claim 5, characterized in that, Feature extraction of image targets includes: inputting images from the image target library into the industrial quality inspection large model, and the output is the image target features.

8. The intelligent data cleaning method according to claim 5, characterized in that, Clustering and distribution visualization analysis includes: the target feature clustering method is to calculate the Mahalanobis distance between all target features, cluster targets with a Mahalanobis distance less than 0.01, and manually confirm their categories based on typical examples of the clustering results; the distribution visualization analysis method is to use t-SNE to visualize the feature data in 2D or 3D.

9. The intelligent data cleaning method according to claim 5, characterized in that, Sparse sample screening includes: defining the sample density as the number of targets within a specified Mahalanobis distance, and screening out samples whose sample density is less than a preset density threshold.

10. The intelligent data cleaning method according to claim 5, characterized in that, Fuzzy and isolated sample filtering includes: representing fuzzy samples as samples whose inter-class distance is less than their intra-class distance, and isolated samples as samples in which only a few sample targets exist within a unit Mahalanobis distance.

11. The intelligent data cleaning method according to claim 5, characterized in that, Data augmentation generation includes: first, applying textures to sparse targets by rotating, scaling, adjusting brightness / contrast, slightly blurring, and adding noise, and then manually confirming the samples to be augmented; next, performing feature comparison in a standard target image library, searching for missing samples in the current data's potential spatial distribution from the standard target image library, generating transfer images, and manually confirming the samples to be augmented.

12. The intelligent data cleaning method according to claim 1, characterized in that: The third-stage standard dataset expansion process includes material category division and data category mapping, category de-obfuscation, data deduplication, data balanced sampling, and data update into the database.

13. The intelligent data cleaning method according to claim 12, characterized in that, Material category classification and data mapping include: first, finding recommended classes in the standard library for each category in the project through feature similarity calculation, and then manually confirming the categories and constructing the mapping.

14. The intelligent data cleaning method according to claim 12, characterized in that, Category de-obfuscation processing includes: projecting the target features of the project data onto a standard data feature library to compare intra-class and inter-class similarities, and calculating the proportion of inter-class similarity greater than intra-class similarity.

15. The intelligent data cleaning method according to claim 12, characterized in that: Data deduplication includes removing duplicates from targets whose features have a cosine similarity greater than a certain threshold.

16. The intelligent data cleaning method according to claim 12, characterized in that, Data equalization sampling includes: using the furthest distance algorithm to target and sample classes with an excessive number of classes.

17. The intelligent data cleaning method according to claim 12, characterized in that: The update process includes new category updates and data updates. When expanding to new categories, labels are incremented to avoid modifying previous data labels. Data updates directly incorporate project data into the standard dataset.

18. A computer-readable storage medium storing computer instructions thereon, characterized in that: The computer instructions execute the intelligent data cleaning method according to any one of claims 1-17.

19. A terminal comprising a memory and a processor, the memory storing a single-task dataset, a standard dataset, an image dataset, and computer instructions executable on the processor, wherein the processor executes the intelligent data cleaning method according to any one of claims 1-17 when executing the computer instructions.

Images