Image data processing method and device, electronic equipment and storage medium
By performing feature extraction, similarity calculation, clustering and deduplication on the drone patrol image dataset, the noise and redundancy problems in the dataset are solved, and the dataset quality and the performance of the target detection model are improved.
Patent Information
- Application Number
- CN202411484621.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-02-07
AI Technical Summary
There is a lot of noise and redundancy in the image dataset captured by the drone during the patrol process, resulting in low quality and consistency of the dataset, affecting the performance of the target detection model.
By acquiring the image dataset and performing feature extraction on each image, the cosine similarity and Euclidean distance between each image and other images are calculated, clustering and deduplication are performed to remove noise and redundant images.
Improves the quality and consistency of the data set, thereby improving the performance and accuracy of the object detection model.
Smart Images
Figure CN119399482A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to technical fields such as data set processing, and in particular to an image data processing method, device, electronic device and storage medium. Background Art
[0002] With the advancement of drone technology, the inspection of distribution lines has gradually changed from traditional manual inspection to drone inspection, which has greatly improved the inspection efficiency. However, drones will take a large number of images on each distribution line, and these images need to be manually labeled with various targets. Due to the huge workload of labeling, it is easy to have problems such as missing and wrong labeling of targets, resulting in a lot of noise in the data set. At the same time, due to the high similarity of images collected by drones, there is a lot of redundancy. Therefore, it is particularly important to clean the data set and remove noise and redundant images. Traditional data cleaning methods usually rely on manual inspection one by one, which is extremely inefficient. Summary of the invention
[0003] To this end, the purpose of the embodiments of the present application is to propose an image data processing method, device, electronic device, storage medium and computer program product to improve the quality and consistency of the data set, thereby improving the performance of the target detection model.
[0004] An embodiment of the present application provides an image data processing method, the method comprising: acquiring an image data set, and performing feature extraction on each image to obtain feature vector data; based on the feature vector data, calculating the cosine similarity between each image and other images, obtaining a first image set based on the cosine similarity, and clustering the image data set based on the feature vector data to obtain a second image set; and deduplicating the image data set based on the first image set and the second image set.
[0005] Exemplarily, clustering the image data set based on the feature vector data to obtain a second image set includes: calculating the Euclidean distance between each image and other images based on the feature vector data; and clustering the image data set based on the Euclidean distance to obtain a second image set.
[0006] Exemplarily, clustering the image data set based on the Euclidean distance to obtain a second image set includes: determining the number of images whose Euclidean distances between the current image and other images are smaller than the Euclidean radius, wherein the Euclidean radius is the average value of the Euclidean distances calculated between all images; clustering the image data set based on the number of images to obtain a second image set.
[0007] Exemplarily, clustering the image data set based on the number of images to obtain the second image set includes: performing preliminary classification on the image data set based on the number of images to obtain a classification result; for core images and boundary images in the classification result, clustering the image data set based on the Euclidean distance between the core images and the Euclidean distance between the boundary images and the core images to obtain multiple image clusters, wherein images whose Euclidean distance between the core images is less than the Euclidean radius are determined to be the same image cluster, and images whose Euclidean distance between the boundary images and the core images is less than the Euclidean radius are determined to be the same image cluster; and determining the image cluster to be the second image set.
[0008] Exemplarily, the preliminary classification of the image data set based on the number of images includes: when the number of images is greater than a preset number threshold, determining that the current image is the core image; when the number of images is less than or equal to the preset number threshold and greater than zero, determining that the current image is the boundary image; when the number of images is zero, determining that the current image is a noise image.
[0009] Exemplarily, the deduplication processing of the image data set based on the first image set and the second image set includes: determining an intersection of the first image set and the second image set, and deleting the intersection.
[0010] Exemplarily, after deduplication processing is performed on the image data set, the method further includes: determining a noise image in the image data set based on the deduplication image data set and a detection model, and cleaning the noise image.
[0011] Exemplarily, determining the noisy image in the image dataset based on the deduplicated image dataset and the detection model includes: dividing the deduplicated image dataset into K sub-datasets, determining K groups of training sets and validation sets corresponding to the training sets based on the K sub-datasets, wherein, for each sub-dataset, other K-1 sub-datasets except its own sub-dataset constitute the training set, and the sub-dataset itself constitutes the validation set, and K is an integer greater than 1; training K detection models based on the K groups of training sets and validation sets to obtain K trained detection models; inferring the validation set based on the trained K detection models to obtain inference results; and determining the noisy image in the image dataset based on the inference results.
[0012] Exemplarily, the image dataset includes annotated position information and annotated category information for a target object, the inference result includes image confidence, inference position information and inference category information for the target object, and determining the noisy image in the image dataset based on the inference result includes: determining the noisy image in the image dataset based on the image confidence, the inference position information, the annotated position information, the inference category information and the annotated category information.
[0013] Exemplarily, the noise image includes at least one of a missing image and an incorrectly labeled image, the annotation position information includes an image annotation box, and the reasoning position information includes an image reasoning box. The determining of the noise image in the image data set based on the image confidence, the reasoning position information, the annotation position information, the reasoning category information and the annotation category information includes: determining that an image whose image confidence is greater than a first preset confidence and whose intersection-and-union ratio between the image reasoning box and the image annotation box is zero is the missing image; determining that an image whose image confidence is greater than a first preset confidence and whose intersection-and-union ratio between the image reasoning box and the image annotation box is greater than a preset threshold and whose reasoning category information is inconsistent with the annotation category information is the incorrectly labeled image; and / or determining that an image whose image confidence is less than a second preset confidence and whose intersection-and-union ratio between the image reasoning box and the image annotation box is greater than a preset threshold is the incorrectly labeled image; wherein the first preset confidence is greater than the second preset confidence.
[0014] Exemplarily, the training set includes image annotation results, and the training of K detection models based on the K groups of training sets and validation sets to obtain K trained detection models includes: iteratively training the K detection models based on the K groups of training sets, and calculating the image gradient of the training set in each iteration; correcting the image annotation results of the training set based on the image gradient, and continuing training based on the modified image annotation results until K trained detection models are obtained.
[0015] Exemplarily, the correcting the image annotation result of the training set based on the image gradient includes: determining that the image whose image gradient is greater than the image gradient mean is the image to be corrected, and determining the inference result of the image to be corrected based on the current detection model; and modifying the image annotation result of the image to be corrected to the inference result.
[0016] Exemplarily, the cleaning the noise image includes: re-labeling the noise image.
[0017] Another embodiment of the present application provides an image data processing device, which includes: an acquisition module, used to acquire an image data set, and perform feature extraction on each image to obtain feature vector data; an acquisition module, used to calculate the cosine similarity between each image and other images based on the feature vector data, obtain a first image set based on the cosine similarity, and perform clustering processing on the image data set based on the feature vector data to obtain a second image set; a processing module, used to perform deduplication processing on the image data set based on the first image set and the second image set.
[0018] Another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any of the above embodiments when executing the computer program.
[0019] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method of any of the above embodiments are implemented.
[0020] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method of any of the above embodiments.
[0021] In the above implementation, an image data set is obtained, and features are extracted for each image to obtain feature vector data; based on the feature vector data, the cosine similarity between each image and other images is calculated, and a first image set is obtained based on the cosine similarity; and based on the feature vector data, the image data set is clustered to obtain a second image set; and the image data set is deduplicated based on the first image set and the second image set. The image data processing method of the present invention can improve the quality and consistency of the data set, thereby improving the performance of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A flowchart of an image data processing method provided in an embodiment of the present application;
[0023] Figure 2 A schematic diagram of deduplication based on cosine similarity and clustering algorithm provided in an embodiment of the present application;
[0024] Figure 3 A flowchart of clustering an image data set provided in an embodiment of the present application;
[0025] Figure 4 A flowchart for calculating the Euclidean distance provided in an embodiment of the present application;
[0026] Figure 5 A flowchart of obtaining an image cluster provided in an embodiment of the present application;
[0027] Figure 6 A flow chart for determining a noisy image in an image data set provided in an embodiment of the present application;
[0028] Figure 7 A flowchart of a training detection model provided for an embodiment of the present application;
[0029] Figure 8 A flow chart of data noise cleaning provided for an embodiment of the present application;
[0030] Fig. 9 A schematic diagram of an image data processing device provided in an embodiment of the present application;
[0031] Fig.10 A block diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION
[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0033] With the advancement of drone technology, the inspection of distribution lines has gradually changed from traditional manual inspection to drone inspection, which has greatly improved the inspection efficiency. However, drones will take a large number of images on each distribution line, and these images need to be manually labeled with various targets. Due to the huge workload of labeling, it is easy to have problems such as missing and wrong labeling of targets, resulting in a lot of noise in the data set. At the same time, due to the high similarity of images collected by drones, there is a lot of redundancy. Therefore, it is particularly important to clean the data set and remove noise and redundant images. Traditional data cleaning methods usually rely on manual inspection one by one, which is extremely inefficient.
[0034] Based on this, this application proposes a data cleaning solution based on an intelligent algorithm, which uses deep learning and image processing technology to automatically detect and correct problems in the data set, improve the quality and consistency of the data, and thus improve the performance of the target detection model.
[0035] Figure 1 It is a flowchart of an image data processing method according to an embodiment of the present application.
[0036] As an example, Figure 1 As shown, the image data processing method includes:
[0037] S101, acquiring an image data set, and performing feature extraction on each image to obtain feature vector data.
[0038] S102, based on the feature vector data, calculating the cosine similarity between each image and other images, obtaining a first image set based on the cosine similarity, and performing clustering processing on the image data set based on the feature vector data to obtain a second image set.
[0039] S103: Perform deduplication processing on the image data set based on the first image set and the second image set.
[0040] Exemplarily, the image data set may be an image of a power distribution circuit taken by a drone. The images collected by the drone have a high similarity and a large redundancy, and the workload of manually removing similar images is large. The present application uses the detection model to extract features from each image in the image data set to obtain feature vector data. That is, each image corresponds to a feature vector. Based on the feature vector data, the cosine similarity of each image with other images is calculated, and the first image set is obtained based on the cosine similarity. It can be understood that the cosine similarity of each image with other images is calculated according to the image feature vector, and similar images are merged together to obtain the first image set. The number of the first image set is multiple. And based on the feature vector data, the image data set is clustered to obtain a second image set. It can be understood that the second image set is an image set after clustering processing. The number of the second image set is also multiple. The image data set is deduplicated based on the first image set and the second image set. For example, the intersection of the first image set and the second image set can be taken as a coincident image, and the intersection image of the first image set and the second image set is discarded.
[0041] The image data processing method of the present application combines the cosine similarity between images and a clustering algorithm to deduplicate the image data set, and the deduplication result is more accurate, thereby improving the quality of the image data set.
[0042] As an example, Figure 2 The schematic diagram of deduplication based on cosine similarity and clustering algorithm is shown. For any original data set of the distribution network, each image in the data set is subjected to feature extraction through a pre-trained model, wherein the pre-trained model can be a resnet50 model pre-trained by imagenet, etc. After the image is inferred by the model, the inference result of the backbone network is taken out and converted into a one-dimensional array (i.e., the feature vector of the image), and then the cosine similarity of the one-dimensional data after inference of each image and the one-dimensional data of other images is calculated. Taking one image as the standard, images with a cosine similarity greater than a threshold with this image constitute a set. For example, Figure 2As shown, based on the model backbone network, the features of picture 1 and picture 2 are extracted, the one-dimensional array is flattened to obtain the feature vector, the similarity between the feature vectors of picture 1 and picture 2 is calculated, and the picture set 1 is obtained. This process is repeated until all similar images in the entire data set are processed. The specific calculation of cosine similarity is as follows:
[0043] y1=f(x1)
[0044] y2=f(x2)
[0045]
[0046] Among them, x1 and x2 represent image 1 and image 2, f(x1) represents the one-dimensional feature of image x1 extracted by the model, f(x2) represents the one-dimensional feature of image x2 extracted by the model, and sim represents the cosine similarity of the two images.
[0047] In addition to calculating the cosine similarity of image features, the present application also considers image deduplication by performing feature clustering on images.
[0048] As an example, Figure 3 As shown, based on the feature vector data, the image data set is clustered to obtain a second image set, including:
[0049] S301, based on the feature vector data, calculating the Euclidean distance between each image and other images.
[0050] S302: Perform clustering processing on the image data set based on Euclidean distance to obtain a second image set.
[0051] Exemplarily, according to the above, each image in the data set is subjected to feature extraction through a pre-trained model, wherein the pre-trained model can be a resnet50 model pre-trained by imagenet, etc. After the image is inferred by the model, the inference result of the backbone network is taken out and converted into a one-dimensional array (i.e., the feature vector of the image), and the Euclidean distance between the feature vector of each image and the feature vector of other images is calculated based on the feature vector data. The image data set is clustered based on the Euclidean distance to obtain a second image set.
[0052] This application combines the cosine similarity and Euclidean distance between images to deduplicate the image data set, and the deduplication result is more accurate and the deduplication effect is better.
[0053] As an example, Figure 4 As shown, clustering is performed on the image data set based on the Euclidean distance to obtain a second image set, including:
[0054] S401, determining the number of images whose Euclidean distances between the current image and other images are smaller than the Euclidean radius, wherein the Euclidean radius is the average value of the Euclidean distances calculated between all images.
[0055] S402: performing clustering processing on the image data set based on the number of images to obtain a second image set.
[0056] Exemplarily, the Euclidean distance between the feature vectors of each image is calculated, and the average value of all calculated Euclidean distances is set as the Euclidean radius, which can be denoted by eps. The number of images whose Euclidean distance between the feature vector of each image and the feature vectors of other images is less than the Euclidean radius eps is counted, and the image data set is clustered based on the number of images to obtain a second image set.
[0057] As an example, Figure 5 As shown, the image data set is clustered based on the number of images to obtain a second image set, including:
[0058] S501, preliminarily classify the image data set based on the number of images to obtain a classification result.
[0059] S502, for the core images and boundary images in the classification results, based on the Euclidean distance between the core images and the Euclidean distance between the boundary images and the core images, clustering the image data set to obtain multiple image clusters, wherein images whose Euclidean distance between the core images is less than the Euclidean radius are determined to be the same image cluster, and images whose Euclidean distance between the boundary images and the core images is less than the Euclidean radius are determined to be the same image cluster.
[0060] S503: Determine that the image cluster is a second image set.
[0061] Exemplarily, based on the number of images whose Euclidean distance between the feature vector of each image and the feature vectors of other images is less than the Euclidean radius eps, the image data set is preliminarily classified to obtain a classification result.
[0062] As an example, a preliminary classification of the image dataset based on the number of images includes:
[0063] When the number of images is greater than a preset number threshold, determining the current image as a core image;
[0064] When the number of images is less than or equal to a preset number threshold and greater than zero, determining that the current image is a boundary image;
[0065] When the number of images is zero, it is determined that the current image is a noise image.
[0066] Exemplarily, the classification result can divide the image into a core image, a boundary image, and a noise image. Set a parameter - a preset number threshold, which can be denoted as P, and the number of images whose Euclidean distances between the current image and other images are less than the Euclidean radius as p. When p>P, the feature points corresponding to the image are marked as core points, and the current image is a core image. If 0<p≤P, the feature points corresponding to the image are marked as boundary points, and the current image is determined to be a boundary image. If p=0, the feature points corresponding to the image are marked as noise points, and the current image is determined to be a noise image.
[0067] Exemplarily, for the core image and the boundary image in the classification result, based on the Euclidean distance between the core images and the Euclidean distance between the boundary image and the core image, the image data set is clustered to obtain multiple image clusters. For example, after the core image is determined, it is determined whether the Euclidean distance between the core images is greater than the Euclidean radius eps, and the core images less than eps are divided into the same image cluster. It can be understood that multiple core images less than eps constitute an image cluster, and an image cluster is a category. The boundary image is divided into different image clusters according to whether the distance from the core image in the current cluster is less than eps. If the boundary image is less than eps according to the distance from the core image in the current cluster, it is determined that the boundary image belongs to the current image cluster. Of course, if the Euclidean distance of a boundary image and the core images in multiple clusters is less than the Euclidean radius, the number of the boundary image and the core image in a cluster whose distance is less than the Euclidean radius can be compared, and the one with more number is divided into this cluster. It can be understood that an image cluster is a category, and the images in each cluster are similar. An image cluster is a second image set.
[0068] For example, the noisy image is directly deleted.
[0069] As an example, performing deduplication processing on the image data set based on the first image set and the second image set includes: determining an intersection of the first image set and the second image set, and deleting the intersection.
[0070] Exemplarily, the cosine similarity result and the clustering result are combined, and the clustered similar picture set and the similar picture set for which the cosine similarity is calculated are intersected to obtain the final similar pictures, which are then deleted from the image set. For example, for an image cluster (the second image set), one picture is used as the standard, and pictures in the cluster are discarded if they are also in the first image set.
[0071] Dataset annotation is also very important in the whole process. However, well-annotated datasets often still have quality problems such as annotation errors, data redundancy, and noisy data, which will seriously affect the performance and reliability of the model. This application also cleans the annotations of the image dataset.
[0072] As an example, after deduplication processing is performed on the image data set, the image data processing method further includes: determining a noise image in the image data set based on the deduplication image data set and the detection model, and cleaning the noise image.
[0073] Exemplarily, each image in the deduplicated image dataset is labeled, and the noise image in the image dataset is determined based on the deduplicated image dataset and the detection model, and the noise image is cleaned. It can be understood that the noise image is found in the deduplicated image dataset based on the detection model, and the noise image represents a noise image that is incorrectly labeled or not labeled. The noise image is found and manually re-labeled. Of course, the noise image can also be labeled based on the detection model.
[0074] The image data processing method of the present application cleans the image data set, finds the noisy images in the image data set, re-labels the noisy images, improves the accuracy of the image data set, and makes subsequent target results more accurate.
[0075] As an example, Figure 6 As shown, the noisy image in the image dataset is determined based on the deduplicated image dataset and the detection model, including:
[0076] S601, divide the deduplicated image dataset into K sub-datasets, and determine K groups of training sets and validation sets corresponding to the training sets based on the K sub-datasets, wherein for each sub-dataset, the other K-1 sub-datasets except the sub-dataset itself constitute the training set, and the sub-dataset itself constitutes the validation set, and K is an integer greater than 1.
[0077] S602: Train K detection models based on K groups of training sets and validation sets to obtain K trained detection models.
[0078] S603, inferring the validation set based on the trained K detection models to obtain inference results.
[0079] S604: Determine a noise image in the image data set based on the inference result.
[0080] Exemplarily, the image data set is first divided into K sub-data sets, which can be numbered 1, 2, 3...K, with a total of K training sets and validation sets, wherein any training set n uses the sub-data set set numbered 1, 2, 3...n-1, n+1..., K as the training set, and the validation set n uses the sub-data set numbered n. It can be understood that for each sub-data set, the other K-1 sub-data sets except its own sub-data set constitute the training set, and its own sub-data set constitutes the validation set, and K is an integer greater than 1. After the K groups of training sets and validation sets are divided, the K training sets are model trained by the detection model to obtain K trained detection models. The validation set is inferred based on the trained K detection models to obtain the inference results. It can be understood that a group of training sets and validation sets corresponds to one detection model. The noise image in the image data set is determined based on the inference results.
[0081] As an example, Figure 7 As shown, the training set includes image annotation results. K detection models are trained based on K groups of training sets and validation sets to obtain K trained detection models, including:
[0082] S701, iteratively train K detection models based on K groups of training sets, and calculate the image gradient of the training set in each iteration.
[0083] S702, correcting the image annotation results of the training set based on the image gradient, and continuing training based on the modified image annotation results until K trained detection models are obtained.
[0084] Exemplarily, during the model training process, K detection models are iteratively trained based on K groups of training sets, and the iteration rounds can be preset, and the training is stopped when the iteration rounds are reached. The image gradient of the training set in each iteration is calculated. The images in the training set include image annotation results, and the image annotation results of the training set are corrected based on the image gradients, and the training is continued based on the modified image annotation results until K trained detection models are obtained.
[0085] As an example, the image annotation results of the training set are corrected based on the image gradient, including: determining that an image whose image gradient is greater than the image gradient mean is an image to be corrected, and determining the inference result of the image to be corrected based on the current detection model; and modifying the image annotation result of the image to be corrected to the inference result.
[0086] Exemplarily, the image gradient of the training set is calculated in each iteration, and the image gradient is an image attribute parameter, that is, each image corresponds to an image gradient. It is determined that the image whose image gradient of the current image is greater than the image gradient mean is the image to be corrected. If the image gradient is greater than the image gradient mean, it means that the annotation result of the image may be wrong. The image is determined to be the image to be corrected, and the reasoning result of the image to be corrected is determined based on the current detection model, that is, the current detection model is used to reason the image to be corrected, and the reasoning result is obtained, and the original image annotation result of the image to be corrected is modified to the reasoning result. It should be noted that the above steps are still in the process of model training. In the process of model training, the original image annotation results of the training data set are adjusted in real time, and training is continued based on the modified annotation results, so that the detection model trained has better performance and is more accurate.
[0087] As an example, during the model training process, each model is trained for N epochs. Epoch can be understood as the iteration round of completely iterating a data set. When the model is trained to the nth epoch, the gradient Grad_l2 of each training image (i.e., the image in the training set) is counted, and the average value of the gradient of all images, Grad_aver, is calculated. The images with gradient values greater than the average gradient in the training images are found. The annotation results of images with gradient values greater than the average gradient are inaccurate. Based on the current model, model inference is performed on images with gradient values greater than the average gradient, and the original annotation results are corrected to the inference results, and then training continues to the end. This method is used to train the K groups of models, and then the K model outputs with the best performance on their respective validation sets are taken. The image gradient calculation process is as follows:
[0088]
[0089] Among them, Grad_l2 represents the gradient calculation of a single image, grad represents the gradient of each learnable parameter in each layer of the model, k represents the number of learnable parameters, Grad_aver represents the average value of the gradient of all training images, and j represents the number of training images.
[0090] After training to obtain K trained detection models, the validation set is inferred based on the trained K detection models to obtain inference results.
[0091] As an example, an image dataset includes annotated position information and annotated category information for a target object, an inference result includes image confidence, inference position information and inference category information for the target object, and determining a noisy image in the image dataset based on the inference result includes:
[0092] A noisy image in an image dataset is determined based on image confidence, inference position information, annotation position information, inference category information and annotation category information.
[0093] Exemplarily, K detection models are respectively subjected to model inference on their respective validation sets to obtain inference results, which include image confidence, inference position information and inference category information for the target object. The image data set includes annotation position information and annotation category information for the target object. The present application determines the noisy image in the image data set based on the inference results and the original annotation results of the image data set.
[0094] As an example, the noisy image includes at least one of a missing label image and a mislabeled image, the annotation position information includes an image annotation box, and the reasoning position information includes an image reasoning box. The noisy image in the image dataset is determined based on the image confidence, the reasoning position information, the annotation position information, the reasoning category information, and the annotation category information, including:
[0095] Determine that an image whose image confidence is greater than a first preset confidence and whose intersection-over-intersection ratio of the image reasoning frame and the image annotation frame is zero is a missing label image;
[0096] Determine as a mislabeled image an image whose image confidence is greater than a first preset confidence, whose intersection-over-union ratio of the image inference frame and the image annotation frame is greater than a preset threshold, and whose inference category information is inconsistent with the annotation category information;
[0097] Determine that an image whose image confidence is less than a second preset confidence and whose intersection-over-intersection ratio of the image reasoning frame and the image annotation frame is greater than a preset threshold is a mislabeled image;
[0098] The first preset reliability is greater than the second preset reliability.
[0099] Exemplarily, K validation sets can be aggregated into a complete data set, and the inference results can be sorted by image confidence. Among them, image confidence is a parameter of the detection model, and the detection model directly outputs the image confidence. Of course, the results with confidence greater than 0.1 in the inference results can also be retained first. The present application sets a first preset confidence and a second preset confidence, wherein the first preset confidence is greater than the second preset confidence. It can be understood that the first preset confidence is the maximum threshold of confidence, which can be recorded as thresh_max, and the second preset confidence is the minimum threshold of confidence, which can be recorded as thresh_min. It can be understood that the image confidence represents the reliability of the image reasoning result, and the intersection and union ratio of the image reasoning box and the image annotation box is the degree of overlap between the image reasoning box and the image annotation box.
[0100] Exemplarily, the inference box whose image confidence is greater than thresh_max in the inference result and whose IOU (Intersection over Union) between the inference box and the annotation box is 0 is screened out. Such an image is likely to be an annotated missing image.
[0101] Exemplarily, the preset threshold may be 0.5. For example, the inference box whose confidence in the inference result is greater than thresh_max and the IOU of the inference box and the annotation box is greater than 0.5, and the image whose inference category information is inconsistent with the annotation category information is a mislabeled image. Among them, the mislabeled image includes the mislabeled category attribute image. For example, the confidence in the inference result is greater than thresh_max, which means that the inference result of the image has a certain degree of reliability, and the IOU of the inference box and the annotation box is greater than 0.5, which indicates that the overlap degree of the inference box and the annotation box is good. Such an inference result should be an accurate inference result. However, the inference category information is inconsistent with the annotation category information, so such an image is a mislabeled category attribute image.
[0102] Exemplarily, images with inference boxes whose confidence is less than thresh_min and annotation boxes whose IOU>0.5 are screened out in the inference results are mislabeled images. The confidence is less than the preset minimum threshold, indicating that the inference result is unreliable. Under normal circumstances, the IOU of the inference box and the annotation box whose confidence is less than the preset minimum threshold should be 0. However, if there is a certain degree of overlap between the inference box and the annotation box, the annotation is mislabeled, and such an image is a mislabeled image.
[0103] As an example, the first preset confidence can be set to 0.6, and the second preset confidence can be set to 0.3. Filter out the inference box whose threshold in the inference result is greater than 0.6 and the image whose IOU with the annotation box is 0. Such images are likely to be images with missing labels; filter out the inference box whose threshold in the inference result is greater than 0.6 and the image whose IOU with the annotation box is greater than 0.5, but the category of the inference result and the annotation result are inconsistent. Such images are likely to be images with mislabeled category attributes; filter out the inference box whose threshold in the inference result is less than 0.3 and the image whose IOU with the annotation box is greater than 0.5. Such images are likely to be images with mislabeled categories.
[0104] As an example, cleaning the noise image includes: re-labeling the noise image.
[0105] Exemplarily, the above rules are used to filter out noisy images, which are cleaned and re-labeled by manual verification. Of course, the detection model can also be used for re-labeling to obtain a data set with better quality.
[0106] Figure 8 This is a flow chart of data noise cleaning according to one embodiment of the present application.
[0107] like Figure 8As shown, for a complete annotated distribution network data set, the data set is first divided into K sub-data sets, numbered 1, 2, 3...K, with a total of K training sets and validation sets, where any training set n uses the sub-data set set numbered 1, 2, 3...n-1, n+1..., K as the training set, and the validation set n uses the sub-data set numbered n. After the K groups of training sets and validation sets are divided, the K data sets are trained by the detection model. The model training can be based on gradient change model training, and the K models with the best performance on each validation set are output. The K models are respectively inferred on their respective validation sets, and the K validation sets are aggregated into a complete data set, and the inference results are sorted by confidence. Two threshold confidence thresholds (first preset confidence and second preset confidence) are set, and noise samples are screened out according to the above method, and these noise samples are cleaned by manual verification.
[0108] The image data processing method of the present application utilizes deep learning and image processing technology to automatically detect and correct problems in the data set, improve the quality and consistency of the data, and thus enhance the performance of the target detection model.
[0109] The present application also proposes an image data processing device.
[0110] As an example, Fig. 9 As shown, the image data processing device includes:
[0111] The acquisition module 901 is used to acquire an image data set and perform feature extraction on each image to obtain feature vector data; the acquisition module 902 is used to calculate the cosine similarity between each image and other images based on the feature vector data, obtain a first image set based on the cosine similarity, and perform clustering processing on the image data set based on the feature vector data to obtain a second image set; the processing module 903 is used to perform deduplication processing on the image data set based on the first image set and the second image set.
[0112] The application also proposes a computer-readable storage medium.
[0113] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned image data processing method are implemented.
[0114] Fig.10 A block diagram of an electronic device provided for an embodiment of the present application.
[0115] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned image data processing method when executing the computer program.
[0116] like Fig.10 As shown, for ease of understanding, the embodiment of the present application shows a specific electronic device.
[0117] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0118] like Fig.10 As shown, the device includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In RAM 1003, various programs and data required for the operation of the electronic device can also be stored. The computing unit 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0119] Multiple components in the electronic device are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0120] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods described above, such as the image data processing method. For example, in some embodiments, the image data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1002 and / or a communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, the image data processing method described above may be executed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the image data processing method in any other appropriate manner (e.g., by means of firmware).
[0121] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this application, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0122] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0123] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0124] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0125] In addition, the terms "first", "second", etc. used in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present application can explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present application, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.
[0126] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific implementation situation.
[0127] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0128] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for processing image data, characterized in that: The method comprises: Obtain an image data set, and perform feature extraction on each image to obtain feature vector data; Based on the feature vector data, calculating the cosine similarity between each image and other images, obtaining a first image set based on the cosine similarity, and performing clustering processing on the image data set based on the feature vector data to obtain a second image set; The image data set is deduplicated based on the first image set and the second image set.
2. The image data processing method according to claim 1, characterized in that: The step of performing clustering processing on the image data set based on the feature vector data to obtain a second image set includes: Based on the feature vector data, calculating the Euclidean distance between each image and other images; The image data set is clustered based on the Euclidean distance to obtain a second image set.
3. The image data processing method according to claim 2, characterized in that: The clustering process is performed on the image data set based on the Euclidean distance to obtain a second image set, including: Determine the number of images whose Euclidean distances between the current image and other images are smaller than the Euclidean radius, wherein the Euclidean radius is the average value of the Euclidean distances calculated between all images; The image data set is clustered based on the number of images to obtain a second image set.
4. The image data processing method according to claim 3, characterized in that: The clustering process is performed on the image data set based on the number of images to obtain a second image set, including: Performing preliminary classification on the image data set based on the number of images to obtain a classification result; For the core images and boundary images in the classification result, clustering the image data set based on the Euclidean distance between the core images and the Euclidean distance between the boundary images and the core images to obtain a plurality of image clusters, wherein the images whose Euclidean distance between the core images is less than the Euclidean radius are determined to be the same image cluster, and the images whose Euclidean distance between the boundary images and the core images is less than the Euclidean radius are determined to be the same image cluster; The image cluster is determined to be the second image set.
5. The image data processing method according to claim 4, characterized in that: The preliminarily classifying the image data set based on the number of images includes: When the number of images is greater than a preset number threshold, determining that the current image is the core image; When the number of images is less than or equal to the preset number threshold and greater than zero, determining that the current image is the boundary image; When the number of images is zero, it is determined that the current image is a noise image.
6. The image data processing method according to claim 1, characterized in that: The performing deduplication processing on the image data set based on the first image set and the second image set includes: An intersection of the first image set and the second image set is determined, and the intersection is deleted.
7. The image data processing method according to claim 1, characterized in that: After performing deduplication processing on the image data set, the method further includes: Based on the deduplicated image data set and the detection model, a noise image in the image data set is determined, and a cleaning process is performed on the noise image.
8. The image data processing method according to claim 7, characterized in that: The determining of the noisy image in the image dataset based on the deduplicated image dataset and the detection model includes: Divide the deduplicated image dataset into K sub-datasets, and determine K groups of training sets and validation sets corresponding to the training sets based on the K sub-datasets, wherein for each sub-dataset, other K-1 sub-datasets except the sub-dataset itself constitute the training set, the sub-dataset itself constitutes the validation set, and K is an integer greater than 1; Training K detection models based on the K groups of training sets and validation sets to obtain K trained detection models; Performing reasoning on the validation set based on the trained K detection models to obtain reasoning results; A noisy image in the image dataset is determined based on the inference result.
9. The image data processing method according to claim 8, characterized in that: The image data set includes annotated position information and annotated category information for a target object, the inference result includes image confidence, inference position information and inference category information for the target object, and determining a noisy image in the image data set based on the inference result includes: A noisy image in the image dataset is determined based on the image confidence, the inferred position information, the labeled position information, the inferred category information, and the labeled category information.
10. The image data processing method according to claim 9, characterized in that: The noise image includes at least one of a missing label image and a mislabeled image, the annotation position information includes an image annotation box, the reasoning position information includes an image reasoning box, and determining the noise image in the image dataset based on the image confidence, the reasoning position information, the annotation position information, the reasoning category information, and the annotation category information includes: Determine that an image whose image confidence is greater than a first preset confidence and whose intersection-over-intersection ratio of the image reasoning frame and the image annotation frame is zero is the missing label image; Determine as the mislabeled image an image whose image confidence is greater than a first preset confidence, whose intersection-over-union ratio of the image inference frame and the image annotation frame is greater than a preset threshold, and whose inference category information is inconsistent with the annotation category information; and / or, Determine that an image whose image confidence is less than a second preset confidence and whose intersection-over-combination ratio of the image reasoning frame and the image annotation frame is greater than a preset threshold is the mislabeled image; The first preset reliability is greater than the second preset reliability.
11. The image data processing method according to claim 8, characterized in that: The training set includes image annotation results, and the K detection models are trained based on the K groups of training sets and validation sets to obtain the trained K detection models, including: Iteratively training K detection models based on the K groups of training sets, and calculating the image gradient of the training sets in each iteration; The image annotation results of the training set are modified based on the image gradient, and training is continued based on the modified image annotation results until K trained detection models are obtained.
12. The image data processing method according to claim 11, characterized in that: The correcting the image annotation result of the training set based on the image gradient includes: Determine that the image whose image gradient is greater than the image gradient mean is the image to be corrected, and determine the inference result of the image to be corrected based on the current detection model; The image annotation result of the image to be corrected is modified to the inference result.
13. The image data processing method according to claim 7, characterized in that: The cleaning process of the noise image comprises: The noisy image is re-labeled.
14. An image data processing device, characterized in that: The device comprises: The acquisition module is used to acquire the image data set and perform feature extraction on each image to obtain feature vector data; an obtaining module, configured to calculate the cosine similarity between each image and other images based on the feature vector data, obtain a first image set based on the cosine similarity, and perform clustering processing on the image data set based on the feature vector data to obtain a second image set; A processing module is used to perform deduplication processing on the image data set based on the first image set and the second image set.
15. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method described in any one of claims 1 to 13 when executing the computer program.
16. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 13 are implemented.