Image processing method and apparatus, image sample classification method and apparatus, and storage medium

Abstract

Provided in the embodiments of the present application are an image processing method and apparatus, an image sample classification method and apparatus, and a storage medium. The processing method comprises: performing feature extraction on a plurality of first images by using a feature extraction network, so as to obtain a plurality of first feature vectors; performing clustering processing on the plurality of first feature vectors, so as to acquire first vector groups and first representative feature vectors; for each first vector group, calculating distribution data of the first vector group; on the basis of respective distribution data of a first preset number of reference vector groups of reference images, correcting distribution data of a plurality of first vector groups; and after correction, using first representative feature vectors of at least some of the first vector groups and the corrected distribution data as feature representation data of the plurality of first images. By using the feature representation data, automatic classification of images of the same type can be quickly and accurately realized. Moreover, the solution has a wider application range.

Description

Image processing method, image sample classification method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 4, 2024, with application number 202410010706.X and invention name “Image processing method and image sample classification method, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and more specifically to an image processing method, an image sample classification method, an image processing device, an image sample classification device, an electronic device, and a storage medium. Background Art

[0003] With the development of machine vision technology, images are needed in many application scenarios to achieve various production purposes.

[0004] For example, in the industrial sector, particularly in high-tech sub-sectors like wafer production, product yield requirements are extremely high. Therefore, there's a desire to leverage product images for defect detection. However, when production volumes are low or mass-produced, the number of defective products is limited. Consequently, the number of defective product images is also limited, making the collection of defective product image sample sets extremely challenging. Due to the limited number of images in a sample set, it's often difficult to perfectly represent the characteristics of defective products, resulting in very low accuracy in automated defect detection.

[0005] Therefore, there is an urgent need for an image processing method to process limited images so that they can more ideally represent the characteristics of the target objects therein. Summary of the Invention

[0006] In order to at least partially solve the problems existing in the prior art, according to the first aspect of the present application, an image processing method is provided, comprising: using a feature extraction network to perform feature extraction on multiple first images respectively to obtain multiple first feature vectors; performing clustering processing on the multiple first feature vectors to obtain a first preset number of first vector groups representing the multiple first images and a first representative feature vector representing each first vector group; for each first vector group, calculating the distribution data of the first vector group, wherein the distribution data includes the weight of the first vector group and / or the distribution variance of the first feature vector in the first vector group; based on the distribution data of each of the first preset number of reference vector groups of the reference image, correcting the distribution data of the multiple first vector groups, wherein the reference vector groups correspond one-to-one to the first vector groups; and using the corrected first representative feature vectors of at least part of the first vector groups and the corrected distribution data as feature representation data of the multiple first images.

[0007] In one possible implementation, the feature extraction network includes two cascaded feature extraction networks.

[0008] In one possible implementation, the distribution data is a weight, and calculating the distribution data of the first vector group includes: calculating the distance between all first eigenvectors in the first vector group and the first representative eigenvector in the first vector group; calculating the ratio between a first number and the total number of the plurality of first eigenvectors as the weight of the first vector group; wherein the first number is the number of first eigenvectors whose distance from the first representative eigenvector in the first vector group is less than a distance threshold.

[0009] In one possible implementation, the distribution data includes weights, and based on the distribution data of each of the first preset number of reference vector groups of the reference image, the distribution data of the multiple first vector groups are corrected, including: selecting a second preset number of weights from the weights of the first preset number of first vector groups, wherein the selected weights are greater than the unselected weights; and using the weights of the first reference vector group to correspondingly replace the selected weights.

[0010] In one possible implementation, the processing method also includes: calculating statistical values ​​of multiple first feature vectors; using a feature extraction network to extract features from each reference image in multiple groups of reference images to obtain multiple reference feature vectors, wherein different groups of reference images belong to different types; for each group of reference images, clustering the reference feature vectors of the group of reference images to obtain a first preset number of second vector groups representing the group of reference images and a second representative feature vector representing each second vector group; for each second vector group, calculating the distribution data of the second vector group; for each group of reference images, calculating the statistical value of the reference feature vector of the group of reference images; based on the statistical values ​​of the multiple first feature vectors and the statistical value of the reference feature vector of each group of reference images, calculating the distance between the multiple first images and each group of reference images; based on the distance between the multiple first images and each group of reference images, selecting a group of reference images from the multiple groups of reference images, and determining the second vector group representing the selected group of reference images as the reference vector group.

[0011] According to another aspect of the present application, a method for classifying image samples is provided, comprising:

[0012] Use the feature extraction network to extract features from image samples to obtain sample feature vectors;

[0013] Based on the sample feature vector and the feature representation data of each group of first images in the multiple groups of first images, the type of the image sample is determined, wherein the feature representation data of each group of first images is obtained using the above-mentioned image processing method, and different groups of first images belong to different types.

[0014] In one possible implementation, determining the type of the image sample based on the sample feature vector and feature representation data of each group of first images in the plurality of groups of first images includes:

[0015] Calculating a first distance between each group of first images and the image sample based on the sample feature vector and feature representation data of each group of first images, and obtaining a plurality of first distances;

[0016] The type of a group of first images corresponding to the minimum value among the multiple first distances is determined as the type of the image sample.

[0017] In a possible implementation, the first image is an image of a product, and the type of the first image is determined according to the type of defect of the product.

[0018] According to another aspect of the present application, an image processing device is provided, comprising:

[0019] A first feature extraction module is used to extract features from the plurality of first images respectively using a feature extraction network to obtain a plurality of first feature vectors;

[0020] a clustering module, configured to perform clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and a first representative feature vector representing each first vector group;

[0021] A first calculation module is configured to calculate, for each first vector group, distribution data of the first vector group, wherein the distribution data includes a weight of the first vector group and / or a distribution variance of a first eigenvector in the first vector group;

[0022] a correction module, configured to correct distribution data of the plurality of first vector groups based on distribution data of respective first preset number of reference vector groups of the reference image, wherein the reference vector groups correspond one-to-one to the first vector groups;

[0023] The second calculation module is configured to use the corrected first representative feature vectors of at least part of the first vector group and the corrected distribution data as feature representation data of the plurality of first images.

[0024] According to another aspect of the present application, there is also provided an image sample classification device, comprising:

[0025] A second feature extraction module is used to extract features from the image sample using a feature extraction network to obtain a sample feature vector;

[0026] A classification module is used to determine the type of the image sample based on the sample feature vector and the feature representation data of each group of first images in multiple groups of first images, wherein the feature representation data of each group of first images is obtained using the above-mentioned image processing method, and different groups of first images belong to different types.

[0027] According to another aspect of the present application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned image processing method and / or the above-mentioned image sample classification method when executed.

[0028] According to another aspect of the present application, a storage medium is further provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned image processing method and / or the above-mentioned image sample classification method when running.

[0029] According to the above technical solution, the distribution data of the first vector groups of the first image is corrected using the distribution data of each of the multiple reference vector groups of the reference image. The corrected distribution data and the first representative feature vector of each first vector group are then used as feature representation data for the multiple first images. Because the corrected distribution data and the first representative feature vector of each first vector group accurately represent the features of the target objects in the multiple first images, these feature representation data can be used to quickly and accurately automatically classify images of the same type. Furthermore, the above solution is applicable even in scenarios with limited first images, extending its applicability to a wider range of scenarios.

[0030] The Summary of the Invention introduces a series of simplified concepts that will be further described in detail in the Detailed Description of the Invention. This Summary of the Application does not intend to limit the key features and essential technical features of the claimed technical solution, nor does it intend to determine the scope of protection of the claimed technical solution.

[0031] The advantages and features of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The following drawings of the present application are used as a part of the present application for understanding the present application. The drawings show the embodiments of the present application and the description thereof, which are used to explain the principle of the present application.

[0033] FIG1 is a schematic flowchart of an image processing method according to an embodiment of the present application;

[0034] FIG2 shows a schematic block diagram of a feature extraction network according to an embodiment of the present application;

[0035] FIG3 shows a schematic flow chart of an image sample classification method according to an embodiment of the present application;

[0036] FIG4 shows a flowchart of an image sample classification method according to another embodiment of the present application;

[0037] FIG5 shows a schematic block diagram of an image processing apparatus according to an embodiment of the present application;

[0038] FIG6 shows a schematic block diagram of an image sample classification device according to an embodiment of the present application;

[0039] FIG7 shows a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, a large amount of details are provided to enable a thorough understanding of the present application. However, it will be appreciated by those skilled in the art that the following description is merely illustrative of preferred embodiments of the present application, and the present application may be implemented without one or more of these details. In addition, in order to avoid confusion with the present application, some technical features well known in the art are not described in detail.

[0041] To at least partially address the above technical issues, according to one aspect of the present application, an image processing method is provided. FIG1 illustrates a schematic flow chart of an image processing method 100 according to one embodiment of the present application. As shown in the figure, the method 100 includes steps S120, S130, S140, S150, and S160.

[0042] Step S110 , performing feature extraction on the plurality of first images respectively using a feature extraction network to obtain a plurality of first feature vectors.

[0043] The first image can be an image of any suitable object, and this application does not limit it. For example, the first image is an image of a product, and the type of the first image is determined according to the type of defect of the product. Applying the method of this application to the detection of products can achieve accurate classification of product defects even when it is difficult to collect defect samples of new products, and greatly reduce labor costs, improve production efficiency, and further improve detection accuracy. In embodiments not shown, the image sample classification method of this application can also be applied to other scenarios, such as face recognition. Therefore, the first image can also be other suitable images, and the type to which the first image belongs can also be a suitable classification type of various objects such as people, landscapes, etc. For ease of description, the following will continue to elaborate on the example that the first image is an image of a product and the type to which the first image belongs is the defect type of the product.

[0044] Take the wafer as an example. The multiple first images can be defect samples of a new wafer product. These first images can be defect images that have been manually labeled with defect categories. The number of first images can be arbitrary and is not limited by this application. It is understandable that the number of defect samples of a new wafer product may be relatively small. The multiple first images can be images labeled as a single defect type. For example, if 50 samples are selected for each defect type, the number of first images is 50. Steps S110 to S150 can be performed for the 50 first images of each defect type.

[0045] The first image can be either a grayscale image or a color image. It can be an image of any suitable size and resolution, or an image that meets preset requirements. The first image can be acquired using any existing or future image acquisition method. The first image can be an original image directly acquired by an image acquisition device, or it can be an image obtained by performing a preprocessing operation on the original image. The preprocessing operation may include all operations that facilitate feature extraction, such as improving the visual effect of the image, increasing the clarity of the image, or highlighting certain features in the image. For example, the preprocessing operation may include denoising operations such as filtering, and may also include image adjustment operations such as the grayscale, contrast, brightness, etc. of the overall image that do not affect feature extraction.

[0046] The feature extraction network according to the embodiments of the present application can be any suitable image feature extraction network, which is not limited by the present application, including but not limited to a convolutional neural network, a residual network, a densely connected network, etc. By way of example and not limitation, the feature extraction network includes two cascaded feature extraction networks. For example, the feature extraction network can be two cascaded convolutional neural networks or two cascaded recurrent neural networks.

[0047] Figure 2 shows a schematic block diagram of a feature extraction network according to one embodiment of the present application. As shown in Figure 2, the feature extraction network can be a portion of a trained feature extraction and classification network. The feature extraction network can include two cascaded ResNet18 feature extraction networks. The feature extraction and classification network consists of the two cascaded ResNet18 feature extraction networks and a classification network. For example, assuming the object is a wafer and the first image is a small batch defect sample of a new product, the feature extraction and classification network can be trained using a large number of defect samples from wafer products. For example, defect data from a large batch of products previously produced or currently in production that has a high similarity to the new wafer product can be used. The defect data can be manually classified and labeled as training samples for the feature extraction and classification network. These large batches of training samples are used to train the entire feature extraction and classification network. It will be appreciated that while these training samples are used to train the entire feature extraction and classification network, the feature extraction network is also trained. This allows the feature extraction network to have better defect feature extraction capabilities for the current wafer product images. For example, the ResNet18_1 network can be pre-trained using the ImageNet image dataset. Only then is the large amount of training samples mentioned above used to train the feature extraction and classification network as a whole, thereby improving the training efficiency and training effect.

[0048] It can be understood that using the above-mentioned trained two cascaded feature extraction networks to perform feature extraction on multiple first images can improve the expressiveness and discrimination ability of the features, thereby improving the accuracy of feature extraction.

[0049] For example, if the size of the first image is 512*512, then the first feature vector can be a 512*1 feature vector. For example, if 50 samples of each defect type are selected and feature extraction is performed on the 50 first images of each defect type, 50 feature vectors can be obtained. These 50 feature vectors can be used as multiple first feature vectors representing the current defect type. Assuming that the total defect sample population has 10 defect types, 10 first feature vector sets can be obtained. Each first feature vector set corresponds to a defect category. Each first feature vector set includes multiple 512*1 first feature vectors.

[0050] Step S120 : performing clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and a first representative feature vector representing each first vector group.

[0051] It is understood that this step can be performed for multiple first feature vectors in each first feature vector set. Clustering is a representative unsupervised classification method that can classify the first feature vectors in each first feature vector set. Any suitable clustering algorithm can be used to implement this step. This includes but is not limited to K-means clustering and hierarchical clustering.

[0052] Take the K-means algorithm as an example. Specifically, first, appropriate cluster centers (representative data points) are selected as initial cluster centers. Distances between first eigenvectors can then be calculated: using a selected distance metric (such as Euclidean distance or cosine similarity), the distance or similarity between each first eigenvector and other first eigenvectors is calculated. Each first eigenvector can then be assigned to the category corresponding to its closest cluster center based on the calculated distance or similarity. Next, for each category, the center or representative point of its first eigenvector can be recalculated, which could be the mean, median, or other value of the first eigenvector points. Iterative optimization is performed to ensure that the cluster center better represents the first eigenvector of the category to which it belongs. The process of assigning first eigenvectors and updating cluster centers is repeated until a termination condition is met, such as a fixed number of iterations or a change in cluster center less than a certain threshold. The final clustering result, i.e., the category to which each first eigenvector belongs, is obtained. Thus, first eigenvectors of the same category can be grouped as a first vector group. Each first vector group has large inter-group differences and small intra-group differences.

[0053] The first preset number can be any suitable value, which can be arbitrarily set according to the total number of samples and the sample distribution. For example, the first preset number can be 5, 4, 3, etc. Take the first preset number of 5 as an example. Assume that each first feature vector set includes 50 first feature vectors. In this step, the K-means clustering algorithm can be used to divide the 50 first feature vectors in each first feature vector set into 5 groups to obtain 5 first vector groups. After the clustering process is completed, the cluster center in each first vector group can be directly used as the first representative feature vector of the first vector group, or the mean vector of each first feature vector contained in each first vector group can be calculated as the first representative feature vector of the first vector group. Thus, each first feature vector set can be divided into 5 first vector groups, and 5 first representative feature vectors corresponding one to one to the 5 first vector groups can be obtained.

[0054] Step S130: For each first vector group, calculate distribution data of the first vector group, wherein the distribution data includes the weight of the first vector group and / or the distribution variance of the first eigenvector in the first vector group.

[0055] Exemplarily, the distribution data includes weights. In the above example, for the first feature vector sets of 50 first images corresponding to each defect category, five first vector groups can be obtained through clustering. The weights of the first vector groups can then be calculated using various suitable methods.

[0056] In one example, the ratio of the number of first eigenvectors in each first vector group to the total number of all first eigenvectors can be directly used as the weight of the first vector group. For example, for a set of first eigenvectors of 50 first images for each defect category, if the number of first eigenvectors in the five clustered first vector groups is 15, 10, 10, 8, and 7, respectively, then the weights of these five first vector groups can be determined to be 0.3, 0.2, 0.2, 0.16, and 0.14, respectively. In another example, calculating the distribution data of the first vector group includes: calculating the distance between all first eigenvectors in the first vector group and the first representative eigenvector in the first vector group; and calculating the ratio between the first number and the total number of the plurality of first eigenvectors as the weight of the first vector group. The first number is the number of first eigenvectors whose distance from the first representative eigenvector in the first vector group is less than a distance threshold. For example, for a first vector group including 20 first eigenvectors, the weight of the first vector group can be determined to be 0.4.

[0057] In other words, a distance threshold can be set to evaluate each first vector group: the calculated distances between all first eigenvectors in the first vector group and the first representative eigenvector in the first vector group are compared with the distance threshold to determine the number of first eigenvectors whose distances are less than the distance threshold, i.e., the first number. Furthermore, the ratio of the first number to the total number of all first eigenvectors can be used to determine the distance threshold. The distance threshold can be arbitrarily set based on the actual sample situation.

[0058] Exemplarily, the distribution data may also include distribution variance, i.e., the variance of the first eigenvectors in the first vector group. It will be appreciated that the variance may reflect the degree of dispersion of the first eigenvectors in each first vector group. In this step, the weight and variance of each first vector group may be calculated separately to serve as the distribution data.

[0059] Step S140: Correcting the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image, wherein the reference vector groups correspond one-to-one to the first vector groups.

[0060] The distribution data of each of the first preset number of reference vector groups may be obtained in advance. The distribution data of each of the reference vector groups may be obtained using any suitable method, which is not limited in this application.

[0061] The reference image and the first image can be images of the same type of product. For example, if the aforementioned 50 first images are defect images of a defect category (e.g., defect A) corresponding to a new wafer product, the reference image in this step can be a defect image of a large-volume wafer product previously produced or currently in production that has a high degree of similarity to the defect characteristics of the new wafer product. In other words, the defect categories of the defects in the reference image and the first image can be similar. For example, the defect category of the reference image can be defect A'.

[0062] Taking the first preset number of 5 as an example, the distribution data of the 5 first vector groups can be corrected based on the distribution data of each of the 5 reference vector groups of the reference image. The distribution data of the 5 first vector groups can also be obtained by the method of steps S110 to S130 above. For example, the reference image is an image of a defect category selected from defect images of a large batch of wafer products. The defect images of a large batch of wafer products can also be reference images that have been manually labeled with defect categories. The reference images of each defect category can include 1000. The feature extraction network can be used to extract features from the 1000 reference images of each defect category to obtain 1000 reference feature vectors. Then, a method similar to steps S120 and S130 above can be used to cluster the reference feature vectors of the 1000 reference images of each defect category into 5 groups of second vector groups, and the distribution data and second representative feature vector of each group of second vector groups can be obtained. Then, various suitable methods can be used to select five second vector groups from 1000 reference images corresponding to a defect category (e.g., defect A') as the five reference vector groups for the 1000 reference images. Calibration is then performed based on the distribution data of these five reference vector groups against the distribution data of the five first vector groups from the 50 first images obtained in step S130.

[0063] The distribution data of the five first vector groups can be corrected using any suitable method. For example, the five reference vector groups can be matched with the five first vector groups, and then the distribution data of the reference vector groups can be used to replace the distribution data of the matched first vector groups. In this step, the distribution data of all first vector groups can be adjusted, or only the distribution data of some first vector groups can be adjusted.

[0064] For example, in the case where the distribution data includes weights and variances, the weights and variances of the matched first vector group may be corrected respectively according to the weights and variances of the reference vector group, or only one of them may be corrected.

[0065] Step S150 : Using the corrected first representative feature vectors of at least part of the first vector group and the corrected distribution data as feature representation data of the plurality of first images.

[0066] The first representative feature vectors and corrected distribution data of at least some of the first vector groups can be stored as feature representation data for the plurality of first images. For example, if the first preset number is five, the first representative feature vectors and corrected distribution data of the five first vector groups can be used as feature representation data for 50 first images. Alternatively, the first representative feature vectors and corrected distribution data of three or four of the five first vector groups can be used as feature representation data for the 50 first images.

[0067] According to the above technical solution, the distribution data of the first vector groups of the first image is corrected using the distribution data of each of the multiple reference vector groups of the reference image. The corrected distribution data and the first representative feature vector of each first vector group are then used as feature representation data for the multiple first images. Because the corrected distribution data and the first representative feature vector of each first vector group accurately represent the features of the target objects in the multiple first images, these feature representation data can be used to quickly and accurately automatically classify images of the same type. Furthermore, the above solution is applicable even in scenarios with limited first images, extending its applicability to a wider range of scenarios.

[0068] Exemplarily, the distribution data includes weights. Correcting the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image includes: selecting a second preset number of weights from the weights of the first preset number of first vector groups, wherein the selected weights are greater than unselected weights; and replacing the selected weights with the weights of the first reference vector group.

[0069] As mentioned above, larger weights correspond to more frequently occurring defect categories. Adjusting the weights for more frequently occurring defects provides a more useful reference. For example, if the first preset number is 5, then five weights corresponding to five first vector groups can be obtained for the multiple first images. These five vector groups can then be sorted by weight, with larger weights first and smaller weights last. Similarly, the five reference vector groups for the reference images are sorted by weight.

[0070] The second preset number is less than or equal to the first preset number. Taking the first preset number of 5 as an example, the second preset number can be any number less than or equal to 5. For example, the second preset number is 3. In this step, the top three first vector groups can be selected from the five first vector groups, and the weights of these three first vector groups can be used as the weights to be corrected. Similarly, the top three reference vector groups can be selected from the five reference vector groups, and the weights of these three reference vector groups can be used as the target weights. Then, the three target weights can be replaced one by one with the three weights to be corrected. For example, the weights of the five reference vector groups are 0.3, 0.2, 0.2, 0.15, and 0.15, respectively, and the weights of the five first vector groups are 0.5, 0.2, 0.15, 0.1, and 0.05, respectively. Then, the weights of the five first vector groups after correction are 0.3, 0.2, 0.2, 0.1, and 0.05, respectively. In an embodiment not shown, the largest two or four weights can also be replaced depending on the situation. In this way, the feature representation data of small sample defect categories with relatively concentrated distribution can be corrected separately, making the final classification result more accurate.

[0071] In other examples, the distribution data includes distribution variances and weights. Correcting the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image may further include: replacing the distribution variances of the second preset number of first vector groups with the distribution variances of the second preset number of reference vector groups, based on the weights of the plurality of first vector groups. For example, the variances of three reference vector groups with larger weights may be replaced with the variances of three first vector groups with larger weights.

[0072] Exemplarily, step S150 uses the corrected first representative eigenvectors of at least a portion of the first vector groups and the corrected distribution data as feature representation data for the plurality of first images, including using the weights and first representative eigenvectors of the first vector groups whose weights have been replaced as feature representation data for the plurality of first images. For example, the weights and first representative eigenvectors of the first three first vector groups can be used as feature representation data for 50 first images.

[0073] Exemplarily, the method 100 further includes step S161 , step S162 , step S163 , step S164 , step S165 , step S166 and step S167 .

[0074] Step S161 calculates the statistical values ​​of multiple first eigenvectors. For example, if the first image is a defective image of a new wafer product, the mean vector of the first eigenvectors of the 50 first images of the current defect category can be calculated to represent the general characteristics of the defect, such as grayscale changes. The variance vector of the first eigenvectors of the 50 first images of the current defect category is calculated to represent the frequency characteristics of the defect, such as changes in high and low frequencies. The covariance of the first eigenvectors of the 50 first images of the current defect category is calculated to represent other attributes of the defect, such as changes in texture and color.

[0075] Step S162 : Using a feature extraction network, extract features from each reference image in the plurality of groups of reference images to obtain a plurality of reference feature vectors, wherein the reference images in different groups belong to different types.

[0076] As mentioned above, a large amount of defect data can be collected and classified and labeled accordingly. For example, the defect data can use the defect data of previously produced large quantities of products, or the defect data of currently produced large quantities of products. For the large amount of defect data collected, the defects can be manually classified and labeled with category labels. The defect samples labeled with the defect category can be used as reference images of different types. For example, 10 categories of defect samples are collected and labeled in advance, and each category of defect samples contains 1,000 images. Then, the 1,000 images of each category of defect samples can be used as a group of reference images. The trained two-cascaded resnet18 feature extraction network shown in Figure 2 can be used to extract features from these 10 groups of reference images to obtain 10,000 reference feature vectors corresponding to the 10,000 reference images.

[0077] As mentioned above, for some defects with obvious differences, such as missing corners and stains, the two categories can be easily distinguished. Therefore, different types of reference images can be grouped, and features can be extracted from each reference image in each group to generate multiple sets of reference feature vectors.

[0078] Step S163 : performing clustering processing on the reference feature vectors of each group of reference images to obtain a first preset number of second vector groups representing the group of reference images and a second representative feature vector representing each second vector group.

[0079] A method similar to the aforementioned step S120 may be used to perform clustering processing on the 1000 reference feature vectors in each group of reference images to obtain five second vector groups and a second representative feature vector of each second vector group.

[0080] Step S164: For each second vector group, calculate the distribution data of the second vector group. This step is similar to the method of the aforementioned step S140 and will not be repeated here.

[0081] Step S165 : For each set of reference images, calculate the statistical values ​​of the reference feature vectors of the set of reference images. For example, calculate the mean vector, variance vector, and covariance of 1000 reference feature vectors of each set of reference images.

[0082] In step S166, based on the statistical values ​​of the plurality of first eigenvectors and the statistical values ​​of the reference eigenvectors of each reference image group, a distance between the plurality of first images and each reference image group is calculated. In step S167, based on the distances between the plurality of first images and each reference image group, a group of reference images is selected from the plurality of reference image groups, and a second vector group representing the selected group of reference images is determined as a reference vector group.

[0083] For example, the correlation coefficient can be calculated based on the statistical values ​​of the first eigenvectors, such as the feature mean, variance, and covariance, and the statistical values ​​of the reference eigenvectors for each set of reference images using the Euclidean distance. The calculation formula is as follows: Y = a(distance<mean>) + b(distance<variance>) + c(distance<covariance>)

[0084] A, b, and c can represent weighting coefficients, which can be set arbitrarily according to actual needs. For example, the values ​​of a, b, and c are 0.5, 0.3, and 0.2, respectively. Exemplarily, the above formula can be used to calculate the distance between the first defect category represented by multiple first images and the second defect category represented by each group of reference images, and a group of reference images of the second defect category corresponding to the minimum distance can be used as calibration reference images. And a first preset number of second vector groups representing the calibration reference images can be used as the corresponding reference vector group. In other words, the defect categories of large-volume wafer products similar to the defect categories of the above-mentioned 50 new wafer products can be determined.

[0085] In this way, the correspondence between the types of multiple first images and each set of reference images can be determined based on statistical data. For multiple sets of first images, a set of reference images that matches (has a similar category) can be determined for each set of first images. Specifically, by matching the second set of vectors with the first set of vectors, a set of reference vectors similar in category to the first set of vectors can be determined. Furthermore, the distribution data of the reference set of vectors can be used to calibrate the distribution data of the first set of vectors.

[0086] According to another aspect of the present application, a method for classifying image samples is also provided. FIG3 shows a schematic flow chart of a method 300 for classifying image samples according to an embodiment of the present application. The classification method 300 includes:

[0087] Step S310: Use a feature extraction network to extract features from the image sample to obtain a sample feature vector. For example, the image sample may be defective image data of a new product to be classified. The trained two-cascaded ResNet18 feature extraction network can be used to extract features from the image sample to obtain a sample feature vector.

[0088] Step S320: Determine the type of the image sample based on the sample feature vector and the feature representation data of each group of first images in the plurality of groups of first images. The feature representation data of each group of first images is obtained using the aforementioned image processing method. Different groups of first images belong to different types. The sample feature vector can be matched with the feature representation data of each group of first images, and the type of the image sample can be determined based on the matching results.

[0089] In industrial production, machine learning is widely used for operations such as product location and defect detection. Machine learning algorithms can be categorized as supervised learning and unsupervised learning. For example, supervised learning detects a specific defect, such as a stain, by comparing the inference results with manually calibrated results to produce a result tailored to the user's needs. Unsupervised learning, on the other hand, allows the user to determine the number of categories, while the machine independently determines the defects, grouping similar defects together. This allows the machine to identify multiple groups of different defect types, with significant variation between groups and less variation within the same group. This allows the machine to automatically group defects and adjust the center of each group based on newly added samples, resulting in more accurate grouping. In other words, as the sample size increases, the sample distribution becomes closer to a normal distribution, making the data less susceptible to outliers and more reliable. However, for small sample sizes, using the unsupervised learning approach described above can significantly affect the accuracy of defect classification due to individual sample values ​​deviating significantly from the mean. The image sample classification method described in the embodiments of this application extracts and trains features from products with a large amount of accumulated data to obtain the distribution of each defect category. This feature distribution is then used to calibrate new, similar, or related products, effectively resolving the problem of poor classification results for small samples in existing technologies. Furthermore, the classification results are more accurate.

[0090] Exemplarily, determining the type of the image sample based on the sample feature vector and feature representation data of each group of first images in the plurality of groups of first images includes: calculating a first distance between each group of first images and the image sample based on the sample feature vector and the feature representation data of each group of first images, and obtaining a plurality of first distances. Determining the type of the group of first images corresponding to the minimum value among the plurality of first distances as the type of the image sample.

[0091] As described above, for example, the first representative eigenvectors of each of the five first vector groups corresponding to a set of first images are A, B, C, D, and E, respectively, and the corresponding corrected weights are 0.3, 0.2, 0.2, 0.1, and 0.05, respectively. The feature representation data may include the weights of some of the first vector groups and the first representative eigenvectors of these first vector groups. For example, the feature representation data of a set of first images includes the first representative eigenvectors A, B, and C and the corresponding weights 0.3, 0.2, and 0.2. In this step, a weighted average method can be used to calculate the first distance between each set of first images and the image sample. If the sample eigenvector of the image sample is x, for example, the first distance between the image sample and the set of first images can be equal to 0.3*(xA)+0.2*(xB)+0.2*(xC). Furthermore, a similar method can be used to calculate the first distance between each set of first images and the image sample. The defect category of the set of first images corresponding to the smallest first distance can be used as the defect category of the current image sample. By calculating and comparing the first distances, the image sample type can be quickly determined. This solution has simple execution logic and less computational complexity, so this classification method has higher classification efficiency and more accurate classification results.

[0092] Figure 4 shows a flowchart of an image sample classification method according to another embodiment of the present application. As shown in the figure, in the first stage, defect data from legacy products can be collected and analyzed. First, a large amount of defect data can be collected and labeled accordingly. Defect data from previously produced, large batches of products, or defect data from currently produced, can be used to collect a large defect dataset A. Defects can be manually classified and labeled with category labels (defect categories may be referred to as legacy defects). Next, a feature extraction and classification network can be trained. The feature extraction and classification network consists of three subnetworks: two ResNet18 feature extraction networks and one classification network. The ResNet18_1 network can use weights pre-trained with Imagenet data. The feature extraction and classification network is trained using the manually classified and labeled defect dataset A in S1. After training, the optimal feature extraction and classification network is obtained. Then, the features of each defect image category in the defect dataset A can be statistically distributed. For example, the defect dataset A includes three sets of reference images corresponding to three defect categories. Specifically, for each defect category, feature extraction is performed using the two trained ResNet18 feature extraction networks to obtain multiple reference feature vectors. Calculate the mean, variance, and covariance of multiple reference feature vectors. Then, cluster these multiple reference feature vectors to obtain five second vector groups and the second representative feature vector, weight, and variance of each second vector group. This statistical information for all old defect categories and the five reference vector groups within each old defect category can then be stored in the current product recipe.

[0093] In the second stage, defect data of new products can be collected and analyzed. For example, 30-50 pieces of defect data can be collected for each type of new product, which can meet the small sample classification requirements of this solution for defect data. For example, the new product defect data set B is manually labeled with 3 defect categories (defect categories can be called new defects). According to the above method, the trained feature extraction network is used to extract features from multiple first images of each new defect to obtain multiple first feature vectors. The mean vector, variance vector and covariance of the multiple first feature vectors are counted. Then, the multiple first feature vectors are clustered to obtain 5 first vector groups. The first representative feature vector, weight and variance of each first vector group are counted. Since the number of samples of new products is small, the obtained feature distribution can be called a sparse feature distribution.

[0094] In the third stage, the feature distribution of the old product can be used to calibrate the feature distribution of the new product. First, the distance between each type of new defect and each type of old defect can be calculated. Specifically, the distance between each type of new defect and each type of old defect can be calculated based on the mean vector, variance vector and covariance of multiple first feature vectors of each type of new defect and the mean vector, variance vector and covariance of multiple reference feature vectors of each type of old defect. The old defect with the smallest distance to each type of new defect is determined as a similar defect, and the five second vector groups corresponding to the similar defects are determined as the reference vector groups of the new defect images of this type. For each type of new defect, the weights or variances of the five first vector groups of the new defect are calibrated using the weights or variances of the corresponding five reference vector groups. For example, the weights of the three reference vector groups with the largest weight values ​​can be directly used to replace the weights of the three first vector groups with the largest weight values. In this way, the feature representation data of each of the three types of new defects are obtained.

[0095] In the fourth stage, the defect image to be classified is inferred and assigned to a specific category. When a defect is detected in a new product, a defect image is fed in. A feature extraction network is used to extract features and generate a sample feature vector. Then, based on the feature representation data and sample feature vectors for each of the three new defect categories, the distance between the current defect image and each new defect is calculated. The new defect category with the smallest distance is determined as the defect category of the current defect image.

[0096] According to another aspect of the present application, an image processing device is also provided. FIG5 shows a schematic block diagram of an image processing device 500 according to an embodiment of the present application. As shown in the figure, the image processing device 500 includes:

[0097] A first feature extraction module 510 is configured to perform feature extraction on each of the plurality of first images using a feature extraction network to obtain a plurality of first feature vectors;

[0098] A clustering module 520 is configured to perform clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and a first representative feature vector representing each first vector group;

[0099] A first calculation module 530 is configured to calculate, for each first vector group, distribution data of the first vector group, wherein the distribution data includes a weight of the first vector group and / or a distribution variance of a first eigenvector in the first vector group;

[0100] a correction module 540 for correcting distribution data of the plurality of first vector groups based on distribution data of respective first preset number of reference vector groups of the reference image, wherein the reference vector groups correspond one-to-one to the first vector groups;

[0101] The second calculation module 550 is configured to use the corrected first representative feature vectors of at least part of the first vector group and the corrected distribution data as feature representation data of the plurality of first images.

[0102] According to another aspect of the present application, an image sample classification device is also provided. FIG6 shows a schematic block diagram of an image sample classification device 600 according to an embodiment of the present application. As shown in the figure, the image sample classification device 600 includes:

[0103] A second feature extraction module 610 is configured to extract features from the image sample using a feature extraction network to obtain a sample feature vector;

[0104] The classification module 620 is used to determine the type of the image sample based on the sample feature vector and the feature representation data of each group of first images in the multiple groups of first images, wherein the feature representation data of each group of first images is obtained using the above-mentioned image processing method, and different groups of first images belong to different types.

[0105] According to another aspect of the present application, an electronic device is also provided. Figure 7 shows a schematic block diagram of an electronic device 700 according to an embodiment of the present application. As shown in the figure, electronic device 700 includes a processor 710 and a memory 720. Memory 720 stores computer program instructions, which, when executed by processor 710, are used to execute the above-described image processing method 100.

[0106] According to another aspect of the present application, a storage medium is also provided. Program instructions are stored on the storage medium, and when executed, the program instructions are used to execute the image processing method 100 described above. The storage medium may include, for example, an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media.

[0107] A person skilled in the art can understand the specific implementation schemes and beneficial effects of the above-mentioned image processing device 500, image sample classification device 600, electronic device 700 and storage medium by reading the above-mentioned descriptions of the image processing method 100 and the image sample classification method 300. For the sake of brevity, they will not be repeated here.

[0108] Example:

[0109] Embodiment 1: An image processing method, comprising:

[0110] Using a feature extraction network to perform feature extraction on the plurality of first images respectively to obtain a plurality of first feature vectors;

[0111] performing clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and a first representative feature vector representing each first vector group;

[0112] For each first vector group, calculating distribution data of the first vector group, wherein the distribution data includes a weight of the first vector group and / or a distribution variance of a first eigenvector in the first vector group;

[0113] Correcting the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image, wherein the reference vector groups correspond one-to-one to the first vector groups; and

[0114] The corrected first representative feature vectors of at least part of the first vector group and the corrected distribution data are used as feature representation data of the plurality of first images.

[0115] Embodiment 2: The method according to embodiment 1, wherein the feature extraction network comprises two cascaded feature extraction networks.

[0116] Embodiment 3: The method according to embodiment 1 or 2, wherein the distribution data is a weight, and calculating the distribution data of the first vector group includes:

[0117] Calculating the distances between all first eigenvectors in the first vector group and the first representative eigenvector in the first vector group;

[0118] A ratio between a first number and a total number of the plurality of first eigenvectors is calculated as a weight of the first vector group; wherein the first number is the number of first eigenvectors whose distance from a first representative eigenvector in the first vector group is less than a distance threshold.

[0119] Embodiment 4: According to the method described in any one of Embodiments 1-3, wherein the distribution data includes weights, and the correcting the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image comprises:

[0120] Selecting a second preset number of weights from the first preset number of weights of the first vector group, wherein the selected weights are greater than the unselected weights;

[0121] The selected weights are correspondingly replaced by the weights of the first reference vector group.

[0122] Example 5: According to the method described in any one of Examples 1-4, the processing method further comprises:

[0123] Calculating statistical values ​​of the plurality of first eigenvectors;

[0124] Using a feature extraction network to extract features from each reference image in the plurality of groups of reference images to obtain a plurality of reference feature vectors, wherein the reference images in different groups belong to different types;

[0125] For each group of reference images, clustering the reference feature vectors of the group of reference images to obtain a first preset number of second vector groups representing the group of reference images and a second representative feature vector representing each second vector group;

[0126] For each second vector group, calculating distribution data of the second vector group;

[0127] For each group of reference images, calculating the statistical value of the reference feature vector of the group of reference images;

[0128] Calculating distances between the plurality of first images and each group of reference images based on statistical values ​​of the plurality of first eigenvectors and statistical values ​​of reference eigenvectors of each group of reference images;

[0129] A group of reference images is selected from the multiple groups of reference images according to the distances between the multiple first images and each group of reference images, and a second vector group representing the selected group of reference images is determined as the reference vector group.

[0130] Example 6: A method for classifying image samples, comprising:

[0131] Use the feature extraction network to extract features from image samples to obtain sample feature vectors;

[0132] Based on the sample feature vector and the feature representation data of each group of first images in the multiple groups of first images, the type of the image sample is determined, wherein the feature representation data of each group of first images is obtained using the image processing method described in any one of Examples 1-5, and different groups of first images belong to different types.

[0133] Embodiment 7: According to the method described in embodiment 6, the step of determining the type of the image sample based on the sample feature vector and the feature representation data of each group of first images in the plurality of groups of first images comprises:

[0134] Calculating a first distance between each group of first images and the image sample based on the sample feature vector and feature representation data of each group of first images, and obtaining a plurality of first distances;

[0135] The type of a group of first images corresponding to the minimum value of the multiple first distances is determined as the type of the image sample.

[0136] Embodiment 8: According to the method described in embodiment 6 or 7, the first image is an image of a product, and the type of the first image is determined according to the type of defect of the product.

[0137] Embodiment 9: An image processing device, comprising:

[0138] A first feature extraction module is used to extract features from the plurality of first images respectively using a feature extraction network to obtain a plurality of first feature vectors;

[0139] a clustering module, configured to perform clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and a first representative feature vector representing each first vector group;

[0140] A first calculation module is configured to calculate, for each first vector group, distribution data of the first vector group, wherein the distribution data includes a weight of the first vector group and / or a distribution variance of a first eigenvector in the first vector group;

[0141] a correction module, configured to correct the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image, wherein the reference vector groups correspond one-to-one to the first vector groups;

[0142] The second calculation module is configured to use the corrected first representative feature vectors of at least part of the first vector group and the corrected distribution data as feature representation data of the plurality of first images.

[0143] Embodiment 10: An image sample classification device, comprising:

[0144] A second feature extraction module is used to extract features from the image sample using a feature extraction network to obtain a sample feature vector;

[0145] A classification module is used to determine the type of the image sample based on the sample feature vector and the feature representation data of each group of first images in the multiple groups of first images, wherein the feature representation data of each group of first images is obtained using the image processing method described in any one of Examples 1-5, and different groups of first images belong to different types.

[0146] Example 11: An electronic device comprising a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the image processing method described in any one of Examples 1-5 and / or the image sample classification method described in any one of Examples 6-8 when the processor is running.

[0147] Example 12: A storage medium having program instructions stored thereon, wherein the program instructions are used to execute the image processing method described in any one of Examples 1-5 and / or the image sample classification method described in any one of Examples 6-8 during runtime.

[0148] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0149] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0151] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0152] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach of the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0153] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0154] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0155] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the image processing device 500 and the image sample classification device 600 according to the embodiments of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0156] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0157] The above description is merely a specific embodiment or illustration of a specific embodiment of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that, Including: Performing feature extraction on multiple first images respectively by using a feature extraction network to obtain multiple first feature vectors; Performing clustering processing on the multiple first feature vectors to obtain a first preset number of first vector groups representing the multiple first images and first representative feature vectors representing each first vector group; For each first vector group, calculating distribution data of the first vector group, where the distribution data includes a weight of the first vector group and / or a distribution variance of the first feature vectors in the first vector group; Based on the distribution data of each of the first preset number of reference vector groups of a reference image, correcting the distribution data of the multiple first vector groups, where the reference vector groups and the first vector groups are in one-to-one correspondence; and Taking the first representative feature vectors of at least some of the corrected first vector groups and the corrected distribution data as feature representation data of the multiple first images.

2. The image processing method according to claim 1, wherein, The feature extraction network includes two cascaded feature extraction networks.

3. The image processing method according to claim 1, wherein The distribution data is a weight, and calculating the distribution data of the first vector group includes: Calculating distances between all the first feature vectors in the first vector group and the first representative feature vector in the first vector group; Calculating a ratio between a first quantity and the total quantity of the multiple first feature vectors as the weight of the first vector group; where the first quantity is the number of first feature vectors whose distances from the first representative feature vector in the first vector group are less than a distance threshold.

4. The image processing method according to claim 1, wherein The distribution data includes a weight, and correcting the distribution data of the multiple first vector groups based on the distribution data of each of the first preset number of reference vector groups of a reference image includes: Selecting a second preset number of weights from the weights of the first preset number of first vector groups, where the selected weights are greater than the unselected weights; Correspondingly replacing the selected weights with the weights of a first reference vector group.

5. The image processing method according to any one of claims 1 to 4, characterized in that, The processing method further includes: Calculating statistical values of the multiple first feature vectors; Performing feature extraction on each reference image in multiple groups of reference images by using a feature extraction network to obtain multiple reference feature vectors, where different groups of reference images belong to different types; For each group of reference images, performing clustering processing on the reference feature vectors of the group of reference images to obtain a first preset number of second vector groups representing the group of reference images and second representative feature vectors representing each second vector group; For each second vector group, calculating distribution data of the second vector group; For each group of reference images, calculating statistical values of the reference feature vectors of the group of reference images; Calculating distances between the multiple first images and each group of reference images according to the statistical values of the multiple first feature vectors and the statistical values of the reference feature vectors of each group of reference images; Selecting a group of reference images from the multiple groups of reference images according to the distances between the multiple first images and each group of reference images, and determining the second vector group representing the selected group of reference images as the reference vector group.

6. An image sample classification method, characterized in that, Including: Performing feature extraction on an image sample by using a feature extraction network to obtain a sample feature vector; Based on the sample feature vector and the feature representation data of each group of first images in multiple groups of first images, determine the type to which the image sample belongs, wherein the feature representation data of each group of first images is obtained by using the image processing method according to any one of claims 1 to 5, and the first images in different groups belong to different types.

7. The image sample classification method according to claim 6, wherein The determining the type to which the image sample belongs based on the sample feature vector and the feature representation data of each group of first images in multiple groups of first images includes: According to the sample feature vector and the feature representation data of each group of first images, calculate the first distance between each group of first images and the image sample, and obtain a plurality of first distances; Determine the type to which the group of first images corresponding to the minimum value among the plurality of first distances belongs as the type to which the image sample belongs.

8. The image sample classification method according to claim 6 or 7, characterized in that The first image is an image of a product, and the type to which the first image belongs is determined according to the type of defect of the product.

9. An image processing apparatus, characterized in that, including A first feature extraction module, configured to respectively perform feature extraction on multiple first images by using a feature extraction network to obtain a plurality of first feature vectors; A clustering module, configured to perform clustering processing on the plurality of first feature vectors to obtain a first preset number of first vector groups representing the plurality of first images and first representative feature vectors representing each first vector group; A first calculation module, configured to calculate the distribution data of each first vector group, wherein the distribution data includes the weight of the first vector group and / or the distribution variance of the first feature vectors in the first vector group; A correction module, configured to correct the distribution data of the plurality of first vector groups based on the distribution data of each of the first preset number of reference vector groups of the reference image, wherein the reference vector groups and the first vector groups correspond one by one; A second calculation module, configured to use the first representative feature vectors of at least some of the corrected first vector groups and the corrected distribution data as the feature representation data of the plurality of first images.

10. An image sample classification device, characterized in that, including: A second feature extraction module, configured to perform feature extraction on the image sample by using a feature extraction network to obtain a sample feature vector; A classification module, configured to determine the type to which the image sample belongs based on the sample feature vector and the feature representation data of each group of first images in multiple groups of first images, wherein the feature representation data of each group of first images is obtained by using the image processing method according to any one of claims 1 to 5, and the first images in different groups belong to different types.

11. An electronic device, comprising a processor and a memory, characterized in that, The memory stores computer program instructions, and when the computer program instructions are run by the processor, they are used to execute the image processing method according to any one of claims 1 to 5 and / or the image sample classification method according to any one of claims 6 to 8.

12. A storage medium, on which program instructions are stored, characterized in that, When the program instructions are running, they are used to execute the image processing method according to any one of claims 1 to 5 and / or the image sample classification method according to any one of claims 6 to 8.

Images