Image data classification method, device and system

Abstract

The present disclosure relates to a classification method, device and system for image data, and to the field of computer technology. The method includes inputting each test image data into a neural network model trained by an original training sample set for classification, determining the image type and membership probability to which each image data belongs; dividing the test image data with membership probability greater than a first threshold into an easy-to-classify data set; adding the test image data in the easy-to-classify data set with a classification accuracy less than or equal to a second threshold and a correct classification result to the original training sample set to generate an expanded training sample set; and training the neural network model using the expanded training sample set to determine the image classification model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for classifying image data, a device for classifying image data, a system for classifying image data, and a non-volatile computer-readable storage medium. Background Art

[0002] The emergence of deep neural networks has greatly advanced image processing technology. Faster speed and higher accuracy have made deep neural networks more widely used in various fields. For example, in practical applications such as medicine, remote sensing, face recognition, and autonomous driving, image processing technology based on deep neural networks can classify and process a large number of images.

[0003] In related technologies, based on the classification standards of business scenarios, training samples of various types of images are collected to train a neural network model; and the trained neural network model is used to classify images. Summary of the invention

[0004] According to some embodiments of the present disclosure, a method for classifying image data is provided, including: inputting each test image data into a neural network model trained by using an original training sample set for classification, and determining the image type and membership probability to which each image data belongs; dividing the test image data with a membership probability greater than a first threshold into an easy-to-classify data set; adding the test image data in the easy-to-classify data set with a classification accuracy less than or equal to a second threshold and a correct classification result to the original training sample set to generate an expanded training sample set; and training the neural network model using the expanded training sample set to determine an image classification model.

[0005] In some embodiments, adding test image data in the easy-to-classify data set whose classification accuracy is less than or equal to a second threshold and whose classification results are correct to the training sample set to generate an expanded training sample set includes: adding test image data in the easy-to-classify data set whose classification accuracy is greater than the second threshold to the training sample set to generate an expanded training sample set.

[0006] In some embodiments, the classification method also includes: inputting each test image data into a neural network model trained using an expanded training sample set for classification; processing the expanded training sample set based on whether the test image data that was misclassified this time belongs to an easy-to-classify data set; and using the processed expanded training sample set to train the neural network model again to update the image classification model.

[0007] In some embodiments, the classification method further includes: using the test image data that is correctly classified this time to expand the expanded training sample set again for re-training the neural network model.

[0008] In some embodiments, inputting each test image data into a trained neural network model for classification includes: inputting a first data set and a second data set in each test image data into a trained neural network model for classification. The first data set includes test image data in the easy-to-classify data set whose classification accuracy is less than or equal to a second threshold and whose classification results are wrong. The second data set includes test image data belonging to a difficult-to-classify data set. The difficult-to-classify data set is determined based on the image type to which the test image data with a membership probability less than or equal to the first threshold belongs.

[0009] In some embodiments, processing the expanded training sample set based on whether the currently misclassified test image data belongs to an easily classified data set includes: when the currently misclassified test image data belongs to the first data set, determining the mislabeled samples that need to be filtered in the expanded training sample set based on the misclassified test image data.

[0010] In some embodiments, according to whether the test image data that is incorrectly classified this time belongs to the easy-to-classify data set, processing the expanded training sample set includes: when the test image data that is incorrectly classified belongs to the difficult-to-classify data set, according to the image type of the test image data that is incorrectly classified this time, obtaining sample data of the corresponding image type, and expanding the expanded training sample set again. The difficult-to-classify data set is determined according to the image type to which the test image data with a membership probability less than or equal to a first threshold belongs.

[0011] In some embodiments, obtaining sample data of corresponding image types and further expanding the expanded training sample set includes: performing data enhancement processing on the test image data that is misclassified this time, obtaining sample data of corresponding image types, and further expanding the expanded training sample set.

[0012] In some embodiments, the classification method further includes: calculating the classification accuracy of the neural network model trained again using the processed expanded training data set. Repeating at least one of the following steps until the classification accuracy is greater than a third threshold: expanding the training sample with the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification result is correct; or processing the training sample according to whether the test image data that was incorrectly classified this time belongs to the easy-to-classify data set.

[0013] In some embodiments, the classification method further includes: using a linear adjustment factor and an exponential adjustment factor set for the membership probability, processing a focal loss function with the membership probability as a variable, and determining an improved loss function for training a neural network model. The linear adjustment factor and the exponential adjustment factor are such that: in the case where any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, the value of the improved loss function corresponding to the membership probability is greater than the value of the focal loss function; in the case where any membership probability is greater than the membership probability of the intersection, the value of the improved loss function corresponding to the membership probability is less than the value of the focal loss function.

[0014] In some embodiments, the improved loss function is based on Sure, is the membership probability, ε is the linear adjustment factor, and γ is the exponential adjustment factor.

[0015] In some embodiments, the classification method further includes: inputting each image data into an image classification model to determine the image type to which each image data belongs.

[0016] In some embodiments, each image data is production line image data of a manufacturing industry, and the image type is a product defect type in the production line image data.

[0017] In some embodiments, the classification method further includes: inputting each image data into an image classification model, and labeling each image data according to the classification result.

[0018] According to some other embodiments of the present disclosure, there is provided an image data classification device, comprising: a classification unit, used to input each image data into a neural network model for classification, and determine the image type and membership probability to which each image data belongs; a determination unit, used to classify test image data with a membership probability greater than a first threshold into an easy-to-classify data set; a processing unit, used to add the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to a second threshold and whose classification result is correct, to a training sample set to generate an expanded training sample set; and a training unit, used to train the neural network model using the expanded training sample set to determine the image classification model.

[0019] In some embodiments, the processing unit adds the test image data with a classification accuracy greater than a second threshold in the easy-to-classify data set to the training sample set to generate an expanded training sample set.

[0020] In some embodiments, the classification unit inputs each test image data into a neural network model trained with an expanded training sample set for classification; the processing unit processes the expanded training sample set according to whether the test image data that is incorrectly classified this time belongs to an easy-to-classify data set; the training unit uses the processed expanded training sample set to train the neural network model again to update the image classification model.

[0021] In some embodiments, the processing unit uses the test image data that is correctly classified this time to expand the expanded training sample set again for re-training the neural network model.

[0022] In some embodiments, the classification unit inputs the first data set and the second data set in each test image data into the trained neural network model for classification. The first data set includes test image data whose classification accuracy is less than or equal to the second threshold value and whose classification result is wrong in the easy-to-classify data set. The second data set includes test image data belonging to the difficult-to-classify data set. The difficult-to-classify data set is determined according to the image type to which the test image data whose membership probability is less than or equal to the first threshold value belongs.

[0023] In some embodiments, when the misclassified test image data belongs to the first data set, the processing unit determines the mislabeled samples that need to be filtered in the expanded training sample set according to the misclassified test image data.

[0024] In some embodiments, when the misclassified test image data belongs to a difficult-to-classify data set, the processing unit obtains sample data of a corresponding image type according to the image type of the misclassified test image data, and further expands the expanded training sample set. The difficult-to-classify data set is determined according to the image type to which the test image data with a membership probability less than or equal to the first threshold belongs.

[0025] In some embodiments, the processing unit performs data enhancement processing on the test image data that is misclassified this time, obtains sample data of the corresponding image type, and further expands the expanded training sample set.

[0026] In some embodiments, the processing unit calculates the classification accuracy of the neural network model trained again using the processed expanded training data set; the processing unit repeatedly performs at least one of the following steps until the classification accuracy is greater than a third threshold: expanding the training sample with the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification results are correct; or processing the training sample based on whether the test image data that was incorrectly classified this time belongs to the easy-to-classify data set.

[0027] In some embodiments, the processing unit uses a linear adjustment factor and an exponential adjustment factor set for the membership probability to process the focal loss function with the membership probability as a variable, and determines the improved loss function for training the neural network model. The linear adjustment factor and the exponential adjustment factor are such that: in the case where any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, the value of the improved loss function corresponding to the membership probability is greater than the value of the focal loss function; in the case where any membership probability is greater than the membership probability of the intersection, the value of the improved loss function corresponding to the membership probability is less than the value of the focal loss function.

[0028] In some embodiments, the improved loss function is based on Sure, is the membership probability, ε is the linear adjustment factor, and γ is the exponential adjustment factor.

[0029] In some embodiments, the classification unit inputs each image data into an image classification model to determine the image type to which each image data belongs.

[0030] In some embodiments, each image data is production line image data of a manufacturing industry, and the image type is a product defect type in the production line image data.

[0031] In some embodiments, the classification unit inputs each image data into an image classification model, and the processing unit labels each image data according to the classification result.

[0032] According to some further embodiments of the present disclosure, there is provided a system for classifying image data, comprising: a classification device for executing the image data classification method in any one of the above embodiments; and an image sensor for acquiring image data.

[0033] According to some further embodiments of the present disclosure, there is provided a device for classifying image data, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method for classifying image data in any one of the above embodiments based on instructions stored in the memory device.

[0034] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the image data classification method in any of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0036] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings:

[0037] Figure 1A flowchart showing some embodiments of the image data classification method disclosed herein;

[0038] Figure 2 Show Figure 1 Flowcharts of some embodiments of steps 120 and 130;

[0039] Figure 3 Show Figure 1 Flowcharts of some embodiments of step 140;

[0040] Figure 4 Schematic diagrams showing some embodiments of the image data classification method disclosed in the present invention;

[0041] Figure 5 A block diagram showing some embodiments of the image data classification apparatus disclosed herein;

[0042] Figure 6 A block diagram showing some other embodiments of the image data classification device disclosed in the present invention;

[0043] Figure 7 A block diagram showing some further embodiments of the image data classification apparatus disclosed herein;

[0044] Figure 8 A block diagram illustrating some embodiments of the image data classification system of the present disclosure. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0046] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0048] Technologies, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the authorization specification.

[0049] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0050] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0051] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: due to the complexity of the classification standards of various business scenarios, the samples of various image types are unevenly distributed, resulting in low classification accuracy of the neural network model.

[0052] In view of this, the present disclosure proposes a technical solution for classifying image data, which can expand the number of samples of image types with low classification accuracy, thereby improving the classification accuracy of the neural network.

[0053] In some embodiments, by taking images of products during the production process (such as the screen production process) and having engineers review them, it is possible to determine the type of defects and resolve damaged products in a timely manner. In this way, it is possible to control product quality, detect product defects, and thus improve product yield.

[0054] In order to solve the technical problem of slow review speed caused by the huge number of product images, for example, a deep neural network model can be used to classify a large number of product images.

[0055] However, on the one hand, the image sample data used for training has high noise and the image type is not clearly distinguished, which often increases the probability of misclassification of the image sample data. Moreover, some image types are similar and difficult to distinguish, which also increases the probability of misclassification of the image sample data.

[0056] This will result in a decrease in the accuracy of image processing technology based on deep neural networks, and will make it impossible to effectively identify the type of product defects.

[0057] On the other hand, due to the uneven distribution of image sample data of various types, in the actual process of collecting image sample data, image sample data of a certain type may be missing or the number is small. In other words, the probability of occurrence of different product defects is different, resulting in the tailing characteristics of the distribution of data of different image categories.

[0058] Moreover, due to the complexity of classification standards in actual business, some image types often contain multiple modes. For example, "oil stain" as an image type can contain multiple modes such as "shallow oil stain" and "circular oil stain". This makes it possible for multiple modes to be unevenly distributed even in the same image type.

[0059] In this way, the uneven distribution of image sample data can cause the neural network model to be unable to recognize some image types.

[0060] On the other hand, the data scale of the classified data set generated after the neural network model classification is very large. Manual data cleaning alone takes a very long time to complete, and it is impossible to effectively locate the problems in the classified data set.

[0061] In response to the above technical problems, the present disclosure proposes an improved solution for the loss function, so that the trained neural network model can better identify and distinguish image types with a small amount of sample data. In addition, a data cleaning technical solution is proposed to effectively filter out noise data in the data set and reduce the workload of reviewing data. Moreover, a mechanism for quickly locating problem data is proposed, which can effectively find data in the data set that is difficult to classify and provide a solution for the next round of data cleaning. For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0062] Figure 1 A flowchart showing some embodiments of the image data classification method of the present disclosure.

[0063] like Figure 1 As shown, the method includes: step 110, classifying the test data; step 120, dividing the easy-to-classify data set; step 130, expanding the training samples; and step 140, training the image classification model.

[0064] In step 110, each test image data is input into the neural network model trained by the original training sample set for classification, and the image type and membership probability of each image data are determined. For example, each image data is a production line image data of the production industry, and the image type is the product defect type in the production line image data. The neural network model can be a VGG (Visual Geometry Group Network) model. The test image data is unlabeled image data, and it can be labeled through the neural network model and classification review processing.

[0065] For example, the amount of test image data is much larger than the amount of data in the original training sample set. Therefore, by classifying the test image data through a neural network model, the efficiency of data labeling can be improved.

[0066] In some embodiments, the original training sample set can be obtained by taking product images during the production process. For example, a number of (eg, 300 to 500) image sample data can be selected from each image category included in the original training sample set to train the neural network model.

[0067] In some embodiments, a focal loss function may be used to train a neural network model. For example, a linear adjustment factor and an exponential adjustment factor set for the membership probability may be used to process a focal loss function with the membership probability as a variable to determine an improved loss function for training the neural network model.

[0068] The linear adjustment factor and the exponential adjustment factor are such that: when any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, the value of the improved loss function corresponding to the membership probability is greater than the value of the focal loss function; when any membership probability is greater than the membership probability of the intersection, the value of the improved loss function corresponding to the membership probability is less than the value of the focal loss function.

[0069] That is to say, the linear adjustment factor and the exponential adjustment factor make the slope of the focus loss function curve steeper. Thus, the loss value of the loss function for data with a smaller membership probability (less than the membership probability of the intersection point) is increased, and the loss value of data with a larger membership probability (greater than the membership probability of the intersection point) is reduced.

[0070] For example, the improved loss function is based on Sure, is the membership probability, ε is the linear adjustment factor (such as 0.15), and γ is the exponential adjustment factor (such as 4). The improved focal loss function is as follows:

[0071]

[0072] α is a balancing factor (such as a value of 0.25), which can reduce the impact of uneven distribution of the number of image types; y is whether the current image belongs to the current image type (such as a value of 0 or 1); γ can smoothly adjust the loss value of the image according to the difficulty of classifying the current image; ε can enhance the model's learning ability for difficult-to-classify data and reduce its sensitivity to classified data.

[0073] In this way, applying the improved focal loss function to the loss layer in the network structure can reduce the loss value of easy-to-classify data and increase the loss value of difficult-to-classify data during model training. Therefore, the improved focal loss function enables the neural network model to have a stronger learning ability for defect data with a small number of samples and special morphology.

[0074] After training a neural network model that can classify images, the model can be used to infer the data to be cleaned in the training sample set.

[0075] In step 120, the test image data with a membership probability greater than a first threshold is classified into an easy-to-classify data set.

[0076] In step 130, the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification result is correct is added to the original training sample set to generate an expanded training sample set.

[0077] For example, multiple accuracy detection modules (which may be multiple engineers, multiple machine learning models, etc.) can be used to calculate the classification accuracy of each image type in the easy-to-classify data set; determine whether the classification accuracy of the image type to which each image data in the easy-to-classify data set belongs is less than or equal to a second threshold; and through review processing, determine whether the current classification result of each image data in the easy-to-classify data set is correct.

[0078] In some embodiments, test image data in the easy-to-classify data set with a classification accuracy greater than a second threshold is added to the training sample set to generate an expanded training sample set.

[0079] In the above embodiment, according to the classification results of the neural network model on each test data, the data with low classification accuracy and correct classification belonging to the easy-to-classify image type are screened out from each test data for expanding the training samples. In this way, the number of samples of the image type with low classification accuracy can be expanded, thereby improving the classification accuracy of the neural network.

[0080] In some embodiments, the Figure 2 The embodiment in implements step 120 and step 130.

[0081] Figure 2 Show Figure 1 Flowchart of some embodiments of steps 120 and 130 in FIG.

[0082] like Figure 2 As shown, step 120 includes: step 1210, obtaining classification results; step 1220, judging whether the degree of membership is greater than a first threshold; and step 1230, dividing the second data set. Step 130 includes: step 1310, judging whether the accuracy of the easy-to-classify set is greater than a second threshold; step 1320, judging whether the classification is correct; step 1330, expanding the training samples; and step 1340, dividing the first data set.

[0083] In step 1210, after each test image data is input into the neural network model, the image type (such as product defect type) and its membership probability of each test image data can be obtained.

[0084] In step 1220, it is determined whether the degree of membership is greater than a first threshold. If it is greater than the first threshold, step 1310 is executed; if it is less than or equal to the first threshold, step 1230 is executed.

[0085] For example, if the membership probability is greater than the first threshold (such as 0.8), it is determined that the corresponding image data has passed the model test and can be well recognized by the model, and it is determined as easy-to-classify data; if the membership probability is less than or equal to the first threshold, it is determined that the corresponding image data cannot be well recognized by the model, and it is determined as difficult-to-classify data. The membership probability is the probability that the image data belongs to the image type.

[0086] In step 1230, the corresponding image data is divided into a second data set. The second data set contains difficult-to-classify data.

[0087] In some embodiments, the first data set includes test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification result is wrong. The second data set includes test image data belonging to the difficult-to-classify data set. The difficult-to-classify data set is determined according to the image type to which the test image data with the membership probability less than or equal to the first threshold belongs.

[0088] In step 1310, it is determined whether the classification accuracy of the image type corresponding to each image data in the easy-to-classify data set is greater than a second threshold. If greater than, step 1330 is executed; if less than or equal to, step 1320 is executed.

[0089] In some embodiments, total sample image data is randomly extracted from each image type included in the easy-to-classify data set, and N accuracy detection modules (which may be N engineers, N machine learning models, etc.) are used to perform classification review processing. The accuracy detection module is only used to detect whether the classification result is correct, and does not correct the wrong classification result.

[0090] For example, for class_k image type, extract total operator_i The image data is pushed to multiple accuracy detection modules for detection. The accuracy detection module operator_i determines whether the pushed image data belongs to class_k and filters out the image data with incorrect classification. The amount of residual data after filtering this type of image data is operatir_i .

[0091] The classification accuracy of class_k judged by operator_i is:

[0092]

[0093] Calculate the average accuracy of all accuracy detection modules for this type of data as the classification accuracy of this type of data:

[0094]

[0095] In some embodiments, image data corresponding to image types with classification accuracy greater than a second threshold may be divided into a high-quality image data set to expand training samples.

[0096] In some embodiments, the image data corresponding to the image type with a classification accuracy less than the second threshold value can be classified into a low-quality image data set, and whether to expand it into the training sample is determined by further judging whether the classification is correct. For example, for image types with a classification accuracy less than 0.9, a number (e.g., 1000 to 2000) of images can be selected from the corresponding image types in the easy-to-classify data set and stored in the low-quality image data set.

[0097] In this way, the training sample set can be cleaned according to the image quality to construct a new training sample set.

[0098] In step 1320, it is determined whether each image data in the easily categorized data set is correctly classified. If the classification is correct, step 1330 is executed; if the classification is incorrect, step 1340 is executed.

[0099] In step 1330, the correctly classified image data in the low-quality data set is expanded into the training sample set. The image data in the high-quality image data set can also be expanded into the training sample set.

[0100] In step 1340, the corresponding image data is divided into a first data set. The first data set includes image data that belongs to a low-quality data set in the easy-to-classify data set and is misclassified, that is, first-time misclassified data.

[0101] In some embodiments, the acquired low-quality data set is pushed to the accuracy detection module for review. The accuracy detection module screens the misclassified image data in the low-quality data set, determines the misclassified image data as the first misclassified data, and expands the correctly classified image data into the training sample set.

[0102] After obtaining the expanded training sample set and dividing the first data set into the second data set, Figure 1 The remaining steps in perform image classification.

[0103] In step 140, the neural network model is trained using the expanded training sample set to determine an image classification model.

[0104] In some embodiments, the Figure 3 Step 140 is implemented in the embodiment of FIG.

[0105] Figure 3 Show Figure 1 Flowchart of some embodiments of step 140 in FIG.

[0106] like Figure 3 As shown, step 140 includes: step 1410, classifying the test data again; step 1420, whether it belongs to the easy-to-classify data set; step 1430, determining the training samples that need to be expanded based on the misclassified data; step 1440, determining the training samples that need to be filtered based on the misclassified data; and step 1450, training the neural network model again.

[0107] In step 1410, each test image data is input into the neural network model trained with the expanded training sample set for classification.

[0108] In some embodiments, the first data set and the second data set in each test image data are input into the trained neural network model for classification.

[0109] For example, the neural network model is updated using the expanded training data set (for example, trained using an improved loss function). The first data set containing the once misclassified data and the second data set containing the hard-to-classify data are input into the updated neural network model for classification.

[0110] In some embodiments, the test data that belongs to the easy-to-classify data set and whose classification accuracy is greater than the second threshold in this classification process is divided into a third data set. The test image data in the third data set is input into the trained neural network model for reclassification processing. According to whether the two classification results are consistent, it is judged whether the neural network model correctly classifies the test image data in the third data set.

[0111] For example, if the two classification results are consistent, the classification is judged to be correct; if the two classification results are inconsistent, the classification is judged to be wrong. In this way, it can be quickly determined whether the labeling result of the neural network model for the image data in the third data set is correct.

[0112] In step 1420, it is determined whether the test image data that is misclassified this time belongs to the easy-to-classify data set. If so, step 1430 is executed; otherwise, step 1440 is executed.

[0113] In some embodiments, the classification result is pushed to the accuracy detection module for review to determine whether the classification result is correct; the image data that is misclassified again is screened; and the image data that is misclassified again is collected as secondary misclassified data.

[0114] In step 1430, the misclassified test image data comes from the difficult-to-classify data set, and the image type of the sample data that needs to be expanded again to expand the training sample set is determined based on the image type of the misclassified test image data.

[0115] In some embodiments, the secondary misclassified data belongs to a difficult-to-classify data set, indicating that the current neural network model cannot recognize these image data. These image data may belong to a new image type (such as a new product defect type or a product with a defect that is difficult to identify).

[0116] Through these image data, we can quickly understand the new and strange product defects that exist in actual business, and then collect similar types of image data from the technical and business levels.

[0117] For example, at the technical level, data augmentation technology can be used to simulate this type of defect to generate image sample data of the corresponding image type for expanding the training sample set.

[0118] For example, at the business level, you can focus on collecting image data of this type of defect during production work, and then expand the training samples of this type.

[0119] In step 1440, the incorrectly labeled samples that need to be filtered in the expanded training sample set are determined based on the incorrectly classified test image data. For example, in this case, the incorrectly classified test image data belongs to the first data set, and the incorrectly labeled samples that need to be filtered can be determined based on the image type of the incorrectly classified test image data.

[0120] In some embodiments, the secondary misclassified data originates from the primary misclassified data, and these image data may be caused by the misclassified data (wrongly labeled samples) existing in the original training sample set.

[0121] These image data can be used to quickly understand the errors in the time-lapse sample set so that data cleaning can be performed.

[0122] In step 1450, the neural network model is trained again using the processed expanded training sample set to update the image classification model.

[0123] In some embodiments, the processed expanded training sample set can be used to train other neural network models for image data classification.

[0124] In some embodiments, the classification accuracy of the neural network model trained again using the processed expanded training data set is calculated. Figure 2 and Figure 3 The steps in are repeated until the classification accuracy is greater than the third threshold. In each repetition process, the parameters of the neural network model with the same network framework are updated.

[0125] In some embodiments, the expanded training sample set can be processed by at least one of the following processing methods: processing the expanded training sample set based on whether the test image data that was incorrectly classified this time belongs to an easy-to-classify data set; or using the test image data that was correctly classified this time to expand the expanded training sample set again.

[0126] For example, at least one of the following steps is repeatedly performed until the classification accuracy is greater than a third threshold: the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification results are correct is used to expand the training sample; and the training sample is processed according to whether the test image data that is incorrectly classified this time belongs to the easy-to-classify data set.

[0127] In some embodiments, each image data is input into an image classification model to determine the image type to which each image data belongs.

[0128] For example, each image data is input into an image classification model, and each image data is labeled according to the classification result. In this way, the image data can be labeled efficiently and previous labeling errors can be located.

[0129] In the above embodiment, a larger and more accurate expanded training sample set is established. The neural network model retrained using the expanded training sample set has better classification capabilities for each image type. The newly trained neural network model reclassifies difficult-to-classify data and once-misclassified data more accurately, reducing the workload of classification review.

[0130] In some embodiments, the determined secondary misclassified data contains some product defects that are difficult to identify, and the problematic data can be quickly located. Furthermore, the technical problem can be solved from the algorithm level or the business level.

[0131] For example, at the algorithm level, weights can be configured for image types corresponding to secondary misclassified data, thereby strengthening the loss value weight of this type during model training; data enhancement operations (such as flipping, translation, splicing, etc.) can be performed on image data of this image type to increase the amount of training sample data for this image type.

[0132] For example, at the business level, it is determined whether the image type (such as product defect type) corresponding to the secondary misclassified data can be ignored. If it cannot be ignored, it is possible to focus on collecting sample data of this image type and increase the amount of training sample data of this image type in the training sample set.

[0133] Figure 4 Schematic diagrams showing some embodiments of the image data classification method disclosed in the present invention.

[0134] like Figure 4As shown, the labeled small original training sample set is input into the neural network model (such as VGG 16 convolutional neural model) as the data to be cleaned. According to whether the membership probability of the classification result obtained by the neural network model is higher than the first threshold, the easy-to-classify data set and the difficult-to-classify data set are divided.

[0135] In some embodiments, the accuracy of each image type in the easy-to-classify data set can be calculated by using the accuracy detection module through random sampling. Low-accuracy data and difficult-to-classify data sets are determined as data to be reviewed; and high-accuracy data are expanded to the original training sample set.

[0136] Through classification accuracy review, the image data with incorrect classification is screened out from the low-accuracy data. The correctly classified image data is expanded to the original training sample set; the image data with incorrect classification is determined as the first misclassified data.

[0137] The neural network model is trained using the expanded training sample set, and the updated neural network model is used to reclassify the first misclassified data and the south classified data. The second misclassified data is obtained, and the cause of the classification error is analyzed according to the source of the second misclassified data, so as to clean the image data.

[0138] In the above embodiment, the workload of image data classification and review is reduced, and the cleaning speed of image data is accelerated. Moreover, the rapid location of problems in the data set is achieved, which accelerates the research and development progress of artificial intelligence projects. Quantifying the classification accuracy of the data set effectively improves this indicator, so that the subsequent intelligent algorithm has more accurate and stable performance in actual scenarios.

[0139] Figure 5 A block diagram showing some embodiments of the apparatus for classifying image data of the present disclosure.

[0140] like Figure 5 As shown, the image data classification device 5 includes a classification unit 51 , a determination unit 52 , a processing unit 53 and a training unit 54 .

[0141] The classification unit 51 inputs each image data into the neural network model for classification, and determines the image type and membership probability to which each image data belongs.

[0142] The determination unit 52 classifies the test image data whose membership probability is greater than the first threshold into an easy-to-classify data set.

[0143] The processing unit 53 adds the test image data whose classification accuracy is less than or equal to the second threshold and whose classification result is correct in the easy-to-classify data set to the training sample set to generate an expanded training sample set.

[0144] The training unit 54 trains the neural network model using the expanded training sample set to determine the image classification model.

[0145] In some embodiments, the processing unit 53 adds the test image data with a classification accuracy greater than a second threshold in the easy-to-classify data set to the training sample set to generate an expanded training sample set.

[0146] In some embodiments, the classification unit 51 inputs each test image data into a neural network model trained with an expanded training sample set for classification; the processing unit 53 processes the expanded training sample set according to whether the test image data that is misclassified this time belongs to an easy-to-classify data set; the training unit 54 uses the processed expanded training sample set to train the neural network model again to update the image classification model.

[0147] In some embodiments, the processing unit 53 uses the test image data that is correctly classified this time to expand the expanded training sample set again for retraining the neural network model.

[0148] In some embodiments, the classification unit 54 inputs the first data set and the second data set in each test image data into the trained neural network model for classification. The first data set includes test image data whose classification accuracy is less than or equal to the second threshold value and whose classification result is wrong in the easy-to-classify data set. The second data set includes test image data belonging to the difficult-to-classify data set. The difficult-to-classify data set is determined according to the image type to which the test image data whose membership probability is less than or equal to the first threshold value belongs.

[0149] In some embodiments, when the misclassified test image data belongs to the first data set, the processing unit 53 determines the mislabeled samples that need to be filtered in the expanded training sample set according to the misclassified test image data.

[0150] In some embodiments, when the misclassified test image data belongs to the difficult-to-classify data set, the processing unit 53 obtains sample data of the corresponding image type according to the image type of the misclassified test image data; the processing unit 53 further expands the expanded training sample set. The difficult-to-classify data set is determined according to the image type to which the test image data with a membership probability less than or equal to the first threshold belongs.

[0151] In some embodiments, the processing unit 53 performs data enhancement processing on the test image data that is misclassified this time to obtain sample data of the corresponding image type; the processing unit 53 further expands the expanded training sample set.

[0152] In some embodiments, the processing unit 53 calculates the classification accuracy of the neural network model trained again using the processed expanded training data set; the processing unit 53 repeatedly executes at least one of the following steps until the classification accuracy is greater than a third threshold: expanding the training sample with the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification result is correct; or processing the training sample according to whether the test image data that was incorrectly classified this time belongs to the easy-to-classify data set.

[0153] In some embodiments, the processing unit 53 uses the linear adjustment factor and the exponential adjustment factor set for the membership probability to process the focal loss function with the membership probability as a variable, and determines an improved loss function for training the neural network model.

[0154] The linear adjustment factor and the exponential adjustment factor are such that: when any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, the value of the improved loss function corresponding to the membership probability is greater than the value of the focal loss function; when any membership probability is greater than the membership probability of the intersection, the value of the improved loss function corresponding to the membership probability is less than the value of the focal loss function.

[0155] In some embodiments, the improved loss function is based on Sure, is the membership probability, ε is the linear adjustment factor, and γ is the exponential adjustment factor.

[0156] In some embodiments, the classification unit 51 inputs each image data into an image classification model to determine the image type to which each image data belongs.

[0157] In some embodiments, each image data is production line image data of a manufacturing industry, and the image type is a product defect type in the production line image data.

[0158] In some embodiments, the classification unit 51 inputs each image data into an image classification model, and the processing unit labels each image data according to the classification result.

[0159] Figure 6 A block diagram showing some other embodiments of the apparatus for classifying image data of the present disclosure.

[0160] like Figure 6 As shown, the image data classification device 6 of this embodiment includes: a memory 61 and a processor 62 coupled to the memory 61, and the processor 62 is configured to execute the image data classification method in any embodiment of the present disclosure based on the instructions stored in the memory 61.

[0161] The memory 61 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0162] Figure 7 A block diagram showing some further embodiments of the apparatus for classifying image data of the present disclosure.

[0163] like Figure 7 As shown, the image data classification device 7 of this embodiment includes: a memory 710 and a processor 720 coupled to the memory 710, and the processor 720 is configured to execute the image data classification method in any one of the aforementioned embodiments based on the instructions stored in the memory 710.

[0164] The memory 710 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0165] The image data classification device 7 may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750, the memory 710, and the processor 720 may be connected, for example, via a bus 760. The input / output interface 730 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, a touch screen, a microphone, and a speaker. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0166] Figure 8 A block diagram illustrating some embodiments of the image data classification system of the present disclosure.

[0167] like Figure 8 As shown, the image data classification system 81 includes: a classification device 81, which is used to execute the image data classification method in any of the above embodiments; and an image sensor 82, which is used to acquire image data.

[0168] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media including but not limited to disk storage, CD-ROM, optical storage, etc., which contain computer-usable program code.

[0169] So far, the image data classification method, image data classification device, image data classification system and non-volatile computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0170] The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0171] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for classifying image data, comprising: Input each test image data into the neural network model trained by the original training sample set for classification, and determine the image type and membership probability to which each image data belongs, wherein the test image data is different from the training samples in the original training sample set; Classify the test image data whose membership probability is greater than a first threshold into an easy-to-classify data set; Add the test image data whose classification accuracy is less than or equal to the second threshold and whose classification result is correct in the easy-to-classify data set to the original training sample set to generate an expanded training sample set; Using the expanded training sample set to train the neural network model to determine an image classification model; The focal loss function with the membership probability as a variable is processed by using the linear adjustment factor and the exponential adjustment factor set for the membership probability, and the improved loss function is determined for training the neural network model. The linear adjustment factor and the exponential adjustment factor make it possible for the value of the improved loss function corresponding to any membership probability to be greater than the value of the focal loss function when any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, and for the value of the improved loss function corresponding to any membership probability to be greater than the membership probability of the intersection when the membership probability is greater than the membership probability of the intersection, to be less than the value of the focal loss function.

2. The classification method according to claim 1, wherein: The step of adding the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification result is correct to the training sample set to generate the expanded training sample set comprises: The test image data with a classification accuracy greater than the second threshold in the easy-to-classify data set is added to the training sample set to generate the expanded training sample set.

3. The classification method according to claim 1, wherein: Also includes: Inputting each test image data into the neural network model trained by using the expanded training sample set for classification; Processing the expanded training sample set according to whether the test image data that is misclassified this time belongs to the easy-to-classify data set; The processed expanded training sample set is used to train the neural network model again to update the image classification model.

4. The classification method according to claim 3, further comprising: The expanded training sample set is expanded again using the test image data that is correctly classified this time to train the neural network model again.

5. The classification method according to claim 3, wherein: The inputting each test image data into the trained neural network model for classification comprises: The first data set and the second data set of each test image data are input into the trained neural network model for classification, the first data set includes test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification results are erroneous, and the second data set includes test image data belonging to a difficult-to-classify data set, and the difficult-to-classify data set is determined according to the image type to which the test image data whose membership probability is less than or equal to the first threshold belongs.

6. The classification method according to claim 5, wherein: The processing of the expanded training sample set according to whether the test image data that is misclassified this time belongs to the easy-to-classify data set includes: In a case where the incorrectly classified test image data belongs to the first data set, incorrectly labeled samples that need to be filtered in the expanded training sample set are determined according to the incorrectly classified test image data.

7. The classification method according to claim 3, wherein: The processing of the expanded training sample set according to whether the test image data that is misclassified this time belongs to the easy-to-classify data set includes: In the case that the misclassified test image data belongs to a difficult-to-classify data set, sample data of the corresponding image type is obtained according to the image type of the misclassified test image data, and the expanded training sample set is expanded again. The difficult-to-classify data set is determined according to the image type to which the test image data whose membership probability is less than or equal to the first threshold belongs.

8. The classification method according to claim 7, wherein: The acquiring of sample data of corresponding image types and further expanding the expanded training sample set comprises: Data enhancement processing is performed on the test image data that is misclassified this time, sample data of the corresponding image type is obtained, and the expanded training sample set is expanded again.

9. The classification method according to claim 3, further comprising: Calculating the classification accuracy of the neural network model trained again using the processed expanded training data set; Repeat at least one of the following steps until the classification accuracy is greater than a third threshold: Expand the training samples by using the test image data in the easy-to-classify data set whose classification accuracy is less than or equal to the second threshold and whose classification results are correct; or The training samples are processed according to whether the test image data that is misclassified this time belongs to the easy-to-classify data set.

10. The classification method according to claim 1, wherein: The improved loss function is based on Sure, is the membership probability, is the linear adjustment factor, is the index adjustment factor.

11. The classification method according to any one of claims 1 to 10, further comprising: Each image data is input into the image classification model to determine the image type to which each image data belongs.

12. The classification method according to claim 11, wherein: The image data are production line image data of a manufacturing industry, and the image type is a product defect type in the production line image data.

13. The classification method according to any one of claims 1 to 10, further comprising: Each image data is input into the image classification model, and each image data is labeled according to the classification result.

14. A device for classifying image data, comprising: A classification unit, used for inputting each image data into a neural network model for classification, and determining the image type and membership probability to which each image data belongs; A determination unit, configured to classify the test image data having a membership probability greater than a first threshold into an easy-to-classify data set; A processing unit, configured to add test image data in the easy-to-classify data set whose classification accuracy is less than or equal to a second threshold and whose classification result is correct, to a training sample set to generate an expanded training sample set, wherein the test image data is different from the training samples in the training sample set; A training unit is used to train the neural network model using the expanded training sample set to determine an image classification model, wherein: The processing unit uses the linear adjustment factor and the exponential adjustment factor set for the membership probability to process the focal loss function with the membership probability as a variable, and determines the improved loss function for training the neural network model. The linear adjustment factor and the exponential adjustment factor ensure that when any membership probability is less than the membership probability of the intersection of the focal loss function and the improved loss function, the value of the improved loss function corresponding to the membership probability is greater than the value of the focal loss function, and when any membership probability is greater than the membership probability of the intersection, the value of the improved loss function corresponding to the membership probability is less than the value of the focal loss function.

15. A classification system for image data, comprising: A classification device, used to perform the image data classification method according to any one of claims 1 to 13; An image sensor is used to acquire image data.

16. A device for classifying image data, comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the image data classification method according to any one of claims 1 to 13 based on instructions stored in the memory.

17. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for classifying image data according to any one of claims 1 to 13 is implemented.

Images