Image processing method, device, electronic device and storage medium

Abstract

The embodiments of the present invention provide an image processing method, device, electronic device and storage medium. The image processing method includes: obtaining multiple images to be processed of a target object; for each of the multiple images to be processed, extracting the first eigenvector matrix of the image to be processed; determining the defect position of the target object in each image to be processed; for each image to be processed, determining the first eigenvector of the image to be processed based on the first eigenvector matrix of the image to be processed and the defect position in the image to be processed; and classifying the multiple images to be processed based on the first eigenvector of each image to be processed. The first eigenvector is the eigenvector corresponding to the defect position of the image to be processed, which can significantly improve the accuracy of image defect classification and improve the classification effect.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically to an image processing method, an image processing device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the improvement of computer processing power, image processing technology is increasingly applied to various scenarios, such as product defect classification in industrial inspection. Existing image-based defect classification technologies are mainly divided into supervised and unsupervised types.

[0003] Supervised image-based defect classification methods rely on a large amount of image sample data with category annotations to train the defect classification model. In actual industrial production, it is difficult to collect a large amount of image sample data of products with different categories of defects. In addition, in fast-paced industrial production, product models are updated quickly. Each time a model is updated, a large amount of labeled image sample data needs to be collected again and the defect classification model needs to be retrained. Therefore, it often cannot meet actual needs.

[0004] Unsupervised image-based defect classification methods do not need to use image sample data with category annotations to train the model. However, the image-based defect classification methods in the prior art have low classification accuracy and poor effect. Summary of the invention

[0005] The present invention has been made in view of the above-mentioned problems.

[0006] According to one aspect of the present invention, there is provided an image processing method, comprising:

[0007] Acquire multiple images of the target object to be processed; for each of the multiple images to be processed, extract the first eigenvector matrix of the image to be processed; determine the defect position of the target object in each image to be processed; for each image to be processed, determine the first eigenvector of the image to be processed according to the first eigenvector matrix of the image to be processed and the defect position in the image to be processed; classify the multiple images to be processed according to the first eigenvector of each image to be processed, wherein the defect types of the target object in different categories of images to be processed are different.

[0008] The first eigenvector is the eigenvector corresponding to the defect position of the image to be processed, rather than the eigenvector of the entire image to be processed. Classifying image defects according to the eigenvector corresponding to the defect position of the image to be processed can significantly improve the accuracy of image defect classification and improve the classification effect.

[0009] Exemplarily, the defect position of the target object in each image to be processed is determined, including: for each image to be processed, obtaining the adjacent pixel points of each pixel in the image to be processed, wherein the adjacent pixel points include the pixel points in the image to be processed that are adjacent to the pixel point; for each pixel in the image to be processed, calculating the significance value of the pixel point based on the adjacent pixel points of the pixel point; determining the defect position in the image to be processed based on the significance value of each pixel point in the image to be processed, wherein the significance value of the pixel point at the defect position is greater than the significance value of the pixel point outside the defect position. It has strong applicability, is not limited by the defect type, and is suitable for determining the defect position of various types of defects

[0010] Exemplarily, for each image to be processed, a pixel in the image to be processed corresponds to an eigenvector in the first eigenvector matrix of the image to be processed; based on the adjacent pixels of the pixel, a significance value of the pixel is calculated, including: based on the second eigenvector and at least one third eigenvector in the first eigenvector matrix of the image to be processed, the cosine similarity between the pixel and the adjacent pixels of the pixel is calculated, wherein the pixel corresponds to the second eigenvector, and the adjacent pixels of the pixel correspond to the third eigenvector respectively; based on the cosine similarity, the significance value of the pixel is calculated. The eigenvector can have more features of the corresponding pixel, thereby improving the accuracy of the significance calculation, and the significance value based on the cosine similarity is simple to calculate, which is convenient for determining the significance value of the pixel.

[0011] Exemplarily, for each image to be processed, a pixel point in the image to be processed corresponds to an eigenvector in the first eigenvector matrix of the image to be processed; based on the adjacent pixels of the pixel point, a significance value of the pixel point is calculated, including: based on the second eigenvector and at least one third eigenvector in the first eigenvector matrix of the image to be processed, the Euclidean distance between the pixel point and the adjacent pixels of the pixel point is calculated as a significance value, wherein the pixel point corresponds to the second eigenvector, and the adjacent pixels of the pixel point correspond to the third eigenvector respectively. The Euclidean distance between the second eigenvector and the third eigenvector is calculated as a significance value. The calculation method is simple and does not require complex calculations.

[0012] Exemplarily, the adjacent pixel points also include the first pixel points in other to-be-processed images other than the to-be-processed image among the multiple to-be-processed images, the first pixel point is a pixel point adjacent to the second pixel point in the other to-be-processed images, and the second pixel point is a pixel point corresponding to the position of the pixel point fij. Thus, when calculating the significance value, the features in all to-be-processed images are included, which improves the robustness of the significance value and further improves the accuracy of determining the defect position.

[0013] Exemplarily, for each image to be processed, the pixel points in the image to be processed correspond to the eigenvectors in the first eigenvector matrix of the image to be processed; according to the first eigenvector matrix of the image to be processed and the defect position in the image to be processed, the first eigenvector of the image to be processed is determined, including: in the first eigenvector matrix of the image to be processed, the eigenvector corresponding to the pixel points at the defect position in the image to be processed is determined; the weighted average of the determined eigenvectors is calculated to obtain the first eigenvector of the image to be processed. The calculation method is simple, the calculation efficiency is high, and it can better represent the characteristics of the defect position of the image to be processed to improve the accuracy of classification.

[0014] Exemplarily, according to the first feature vector of each image to be processed, multiple images to be processed are classified, including clustering all images to be processed according to the first feature vector of each image to be processed to obtain a clustering result. In this way, the accuracy of image defect classification can be significantly improved, and the classification effect can be improved.

[0015] Exemplarily, clustering all the images to be processed according to the first eigenvector of each image to be processed to obtain a clustering result includes: concatenating all the first eigenvectors to obtain a second eigenvector matrix; clustering all the images to be processed according to the second eigenvector matrix to obtain a clustering result. Thus, clustering calculation is facilitated and the processing speed of the image processing method is improved.

[0016] Exemplarily, the method further includes: displaying classification results of the plurality of images to be processed, and intuitively displaying the classification results to improve user experience.

[0017] According to another aspect of the present invention, an image processing device is also provided, comprising: an image acquisition module, a feature extraction module, a first determination module, a second determination module and a classification module. The image acquisition module is used to acquire multiple images to be processed of the target object; the feature extraction module is used to extract the first feature vector matrix of the image to be processed for each of the multiple images to be processed; the first determination module is used to determine the defect position of the target object in each image to be processed; the second determination module is used to determine the first feature vector of the image to be processed for each image to be processed based on the first feature vector matrix of the image to be processed and the defect position in the image to be processed; the classification module is used to classify the multiple images to be processed based on the first feature vector of each image to be processed, wherein the defect types of the target object in different categories of images to be processed are different.

[0018] According to another aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned image processing method when the processor is executed.

[0019] According to yet another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the image processing method as described above when running.

[0020] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions, which are used to execute the image processing method as described above when running.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention.

[0023] Figure 1 A schematic flow chart of an image processing method according to an embodiment of the present invention is shown;

[0024] Figure 2 A process diagram of extracting a first eigenvector matrix according to an embodiment of the present invention is shown;

[0025] Figure 3 A schematic diagram of determining a first eigenvector according to an embodiment of the present invention is shown;

[0026] Figure 4 A diagram showing classification results according to an embodiment of the present invention is shown;

[0027] Figure 5 A schematic block diagram of an image processing device according to an embodiment of the present invention is shown;

[0028] Figure 6 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more obvious, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

[0030] Existing unsupervised image classification methods focus more on natural images. The general idea is to rely on deep neural networks to extract global features of the image, and then use traditional clustering algorithms to cluster based on the extracted features. However, there are two differences between natural images and images of defective products. 1) In natural images, the target is generally located in the center of the image, while images of defective products cannot guarantee that the defect is located in the central area. 2) In natural images, the target size occupies a major part of the entire image size, while the defects in images of defective products are generally very small. These two differences result in general unsupervised clustering methods for natural images having poor results on industrial defect data.

[0031] In order to at least solve the above technical problems, the present invention provides an image processing method. The image processing method can be used in any scenario where defects of a target object need to be classified based on an image. In this solution, the image to be processed is classified according to the feature vector corresponding to the defect position of the image to be processed, thereby achieving high-precision image defect classification.

[0032] Figure 1 FIG. 1 is a schematic flow chart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method includes: step S1100, step S1200, step S1300, step S1400 and step S1500.

[0033] Exemplarily, in step S1100 , a plurality of to-be-processed images of the target object are obtained. Specifically, K to-be-processed images of the target object are obtained, where K is a positive integer greater than 1.

[0034] The target object may be any object. Specifically, it may include a wafer, a liquid crystal panel, etc. The image to be processed may be obtained by photographing the target object through a camera device, and the target object may be defective. In some embodiments, K target objects with defects may be photographed by a camera device to obtain an image to be processed corresponding to each target object, totaling K images to be processed. In other embodiments, the camera device may take images of multiple target objects with or without defects, determine the defective images through manual recognition or artificial intelligence algorithms, and use the K images with defects as images to be processed.

[0035] Optionally, in some specific embodiments, the acquired K images to be processed may be images obtained by preprocessing the captured images, such as size standardization, grayscale conversion, histogram equalization, etc., to eliminate the influence of irrelevant factors on subsequent image processing.

[0036] Exemplarily, in step S1200, for each of the multiple images to be processed, a first eigenvector matrix of the image to be processed is extracted. Specifically, for each of the K images to be processed, P k , extract the image to be processed P k The first eigenvector matrix of , where k is a positive integer less than or equal to K.

[0037] Each image P to be processed can be processed by a deep neural network model k Processing is performed to obtain the image P to be processed k The corresponding first eigenvector matrix. The deep neural network model can be trained by a public data set. k In the process of processing, the spatial dimension can be reduced and the channel dimension can be increased to map the channel dimension to the specified dimension and obtain the feature map. The spatial dimension of the feature map can be restored to the image to be processed P k The specified dimension of the channel dimension can be based on the image to be processed P k Thus, each eigenvector in the first eigenvector matrix obtained can be respectively related to the image P to be processed. k The pixel correspondence.

[0038] Figure 2 FIG. 4 shows a process diagram of extracting the first eigenvector matrix according to an embodiment of the present invention. Figure 2 As shown, the image to be processed with a channel dimension of C0 and a spatial dimension of H×W is processed by a deep neural network model with multiple layers. The channel dimension of the feature map corresponding to the image to be processed in the first layer is C1, and the spatial dimension is H / 2×W / 2. The channel dimension of the feature map corresponding to the image to be processed in the second layer is C2, and the spatial dimension is H / 4×W / 4. The channel dimension of the feature map corresponding to the image to be processed in the third layer is C3, and the spatial dimension is H / 8×W / 8. The channel dimension of the feature map corresponding to the image to be processed in the nth layer is C4, and the spatial dimension is H / 2 n ×W / 2 n It can be understood that the channel dimension of the feature map of the image to be processed is C4 and the spatial dimension is H / 2. n ×W / 2 nThe spatial dimension of the feature map of the image to be processed can be restored to H×W by interpolation, and the pixels in the image to be processed have a corresponding relationship with the pixels in the feature map. Therefore, the pixels in the image to be processed can correspond to the feature vector composed of the corresponding pixels in each feature map finally obtained, and the dimension of the feature vector is the channel dimension C4. All feature vectors together constitute the first feature vector matrix.

[0039] Exemplarily, in step S1300, the defect position of the target object in each image to be processed is determined. Specifically, each image to be processed P is determined k The defect location of the target object in the image.

[0040] In some embodiments, the image to be processed P k The defect position in the image to be processed may be marked by a rectangular frame, a binary mask, etc. For example, the position selected by the rectangular frame may be the defect position, or the position where 1 is located in the binary mask may be the defect position. k The defect position of the target object may not be marked. The image P to be processed can be determined by existing or future developed defect detection technologies such as template matching and threshold segmentation. k The defect location in .

[0041] Exemplarily, in step S1400, for each image to be processed, the first eigenvector of the image to be processed is determined according to the first eigenvector matrix of the image to be processed and the defect position in the image to be processed. Specifically, for each image to be processed P k , according to the image to be processed P k The first eigenvector matrix and the image to be processed P k The defect position in the image to be processed is determined k The first eigenvector of .

[0042] As mentioned above, each pixel in the image to be processed corresponds to a eigenvector in the first eigenvector matrix. k , and then determine the eigenvector in the first eigenvector matrix corresponding to the pixel. It can be understood that there may be multiple pixels of the image to be processed corresponding to the defect position. Further, the defect position may correspond to multiple eigenvectors in the first eigenvector matrix. Multiple eigenvectors may be processed to determine the first eigenvector of the image to be processed. For example, multiple eigenvectors may be spliced ​​or weighted to form a eigenvector, and the eigenvector may be used as the first eigenvector. Thus, the image P to be processed may be determined. k The first eigenvector of . Figure 3FIG. 4 is a schematic diagram showing a method for determining a first eigenvector according to an embodiment of the present invention. Figure 3 As shown, the image to be processed P k After the pre-trained model extracts features, the first eigenvector matrix is ​​obtained. The position of the self-color block in the defect position is the position of the defect in the image to be processed. For simplicity, the white block only represents a pixel point in the defect position. According to the pixel point corresponding to the white block, a eigenvector in the first eigenvector matrix can be determined, and the eigenvector can be determined as the first eigenvector.

[0043] Exemplarily, in step S1500, a plurality of images to be processed are classified according to the first feature vector of each image to be processed, wherein the defect types of the target objects in the images to be processed of different categories are different. k The respective first eigenvectors are used to classify the K images to be processed, wherein the defect types of the target objects in the images to be processed of different categories are different.

[0044] Unsupervised classification methods can be used to classify each image P to be processed k It can be understood that the first feature vector can represent the characteristics of defects in the image to be processed, so the classification result of the first feature vector can be used as the classification result of the corresponding K images to be processed. For example, all the first feature vectors can be clustered using methods such as K-means and hierarchical clustering, and each cluster cluster can be a classification result. The cluster cluster corresponding to each first feature vector can be used to determine the classification result of each image P to be processed. k The type of defect corresponding to the target object.

[0045] In the above technical solution, the first eigenvector of the image to be processed is determined based on the first eigenvector matrix of the image to be processed and the defect position of the target object in the image to be processed, and all the images to be processed are classified according to the first eigenvector of each image to be processed. The first eigenvector is the eigenvector corresponding to the defect position of the image to be processed, rather than the global eigenvector of the image to be processed. Classifying image defects according to the eigenvector corresponding to the defect position of the image to be processed can significantly improve the accuracy of image defect classification and improve the classification effect.

[0046] For example, step S1300 determines each image P to be processed k The defect position of the target object in the image P includes: k Execute the following steps S1310, S1320 and S1330.

[0047] In step S1310, the image to be processed P is obtained. kThe neighboring pixels of each pixel fjj in the image to be processed include k In the pixel f ij Adjacent pixel points, where i is less than or equal to the image to be processed P k The width of j is a positive integer less than or equal to the width of the image to be processed P. k It can be understood that the image to be processed P k The product of the width and length of the image P to be processed can be k For example, the image to be processed P k The height and width of the image P to be processed can be 30. k The height and width of the image P are composed of 30 pixels respectively. k A total of 900 pixels are included. In some embodiments, the image to be processed P k In the pixel f ij The adjacent pixels include the pixel f ij In some other embodiments, the pixel points adjacent to the pixel point f in the image to be processed Pk are ij The adjacent pixels also include the pixel f ij In other words, the adjacent pixel points may include the adjacent pixel points in eight directions: front, back, left, right, left front, right front, left back, and right back. ij Alternatively, the adjacent pixel points may include the pixel points that are one pixel apart. ij i is less than or equal to the pixel point of the image to be processed P k The width of j is a positive integer less than or equal to the width of the image to be processed P. k A positive integer of the height of , which can be used to traverse the image P to be processed by adjusting the size of i and j k Each pixel point f ij , to obtain each pixel f ij The neighboring pixels.

[0048] In step S1320, for the image P to be processed k Each pixel point f ij , according to the pixel point f ij The adjacent pixel points of the pixel point are calculated ij In some embodiments, the significance value of the pixel point f ij The distance between the pixel point and its adjacent pixels is used to determine the pixel point f ijSpecifically, each pixel can have multi-channel values, such as RGB three-channel values. Therefore, each pixel can correspond to a vector, which can be calculated by ij The mean of the distances between the vector of the pixel and each adjacent pixel. The larger the mean of the distances, the closer the pixel point f is. ij The greater the significance value of the pixel, the greater the significance value of the pixel. ij The nearest pixel point can be a pixel point that is one pixel away from the pixel point.

[0049] In step S1330, according to the image to be processed P k Each pixel f ij The significance value of the image to be processed P is determined k The defect position in the image is obtained by averaging the pixel points at the defect position, where the pixel point significance value at the defect position is greater than the pixel point significance outside the defect position. It can be understood that the defect is an abnormal part in the target object, which can be expressed as a feature different from the surrounding in the image to be processed, such as different color, brightness, contrast, texture, etc. Therefore, the pixel point significance value at the defect position is greater than the pixel point significance value outside the defect position. The image to be processed P k Middle pixel f ij The pixel points whose significance values ​​are greater than the threshold are determined as the pixel points in the defect position, and the defect position is determined according to the pixel points in the defect position.

[0050] The above technical solution, according to each pixel point f ij The defect location is determined by comparing the significance value of its adjacent pixels. It has strong applicability and is not limited by the defect type. It is suitable for determining the defect location of various types of defects.

[0051] Optionally, in step S1330, according to the image to be processed P k Each pixel f ij The significance value of the image to be processed P is determined k After the defect position is determined, the SAM (Segment Anything Model) model can be used to process the image to be processed with the defect position determined, so as to further improve the accuracy of the defect position.

[0052] For example, each image P to be processed k , the image to be processed P k The pixels in the image P are respectively k The eigenvector in the first eigenvector matrix of corresponds to . Thus, it can be determined that the image to be processed P k Step S1320 calculates the eigenvector in the first eigenvector matrix corresponding to each pixel point in the pixel.ij The adjacent pixel points of the pixel point are calculated ij The significance value of the method includes step S1321 and step S1322.

[0053] In step S1321, according to the image P to be processed k The second eigenvector and at least one third eigenvector in the first eigenvector matrix are used to calculate the pixel point f ij With the pixel f ij The cosine similarity between adjacent pixels of ij Corresponding to the second eigenvector, the pixel point f ij The adjacent pixel points of the image P to be processed correspond to the third feature vector. k The pixel point can be compared with the image to be processed P k In some other embodiments, the image to be processed P k Multiple pixels in the image P to be processed can be k The first eigenvector of corresponds to an eigenvector in the matrix. k It can be divided into multiple regions, each region includes multiple pixels, and each region can correspond to an eigenvector in the first eigenvector matrix. k The pixel point in may have a corresponding relationship with the eigenvector in the first eigenvector matrix. ij The corresponding relationship between the eigenvectors of the first eigenvector matrix and the eigenvectors of the first eigenvector matrix is ​​used to determine an eigenvector of the first eigenvector matrix corresponding to the eigenvector, as the second eigenvector. Similarly, the corresponding third eigenvector can be determined based on the corresponding relationship between each adjacent pixel point and the eigenvectors in the first eigenvector matrix. It can be understood that the pixel point f ij There is at least one adjacent pixel point, so there is at least one corresponding third eigenvector. The cosine similarity between the second eigenvector and each third eigenvector can be calculated respectively.

[0054] In step S1322, the pixel point f is calculated based on the cosine similarity. ij The cosine similarity usually takes a value range of [-1, 1]. The closer the cosine similarity is to 1, the closer the angle between the second eigenvector and the third eigenvector is to 0 degrees, that is, the more similar the two vectors are, the angle is equal to 0, that is, the two vectors are equal. The closer the cosine similarity is to -1, the closer the angle between the second eigenvector and the third eigenvector is to 180 degrees, that is, the less similar the two vectors are, the angle is equal to 180, that is, the two vectors are opposite. In other words, the closer the cosine similarity is to 1, the lower the significance, and the closer it is to -1, the higher the significance.

[0055] From this, the significance value can be calculated according to the following formula:

[0056]

[0057] S ij Represents pixel f ij The significance value of f∈N p (f ij ) represents pixel f ij N adjacent pixels f;

[0058] <f ij , f> means f ij The cosine similarity between the corresponding second eigenvector and the third eigenvector corresponding to the adjacent pixel point f. Therefore, the value range of the significance value can be [0, 2]. The closer the significance value is to 2, the higher the significance of the pixel point. The pixel point with a significance value greater than the threshold can be determined as the pixel point in the defect position to determine the defect position.

[0059] The above technical solution calculates the cosine similarity between the second eigenvector in the first eigenphasor matrix corresponding to the pixel and the third eigenvector in the first eigenvector matrix corresponding to its adjacent pixel, and then determines the significance value of the pixel. The eigenvector can have more features of the corresponding pixel, thereby improving the accuracy of the significance calculation, and the significance value based on the cosine similarity is simple to calculate, which is convenient for determining the significance value of the pixel.

[0060] For example, for each image P to be processed k , the image to be processed P k The pixels in the image P are respectively k The eigenvector in the first eigenvector matrix of corresponds to . Thus, it can be determined that the image to be processed P k Step S1320, based on the pixel point f ij The adjacent pixel points of the pixel point are calculated ij The significance value of the image to be processed P includes: step S1323, according to the image to be processed P k The second eigenvector and at least one third eigenvector in the first eigenvector matrix are used to calculate the pixel point f ij With the pixel f ij The Euclidean distance between adjacent pixels is used as the significance value, where the pixel f ij Corresponding to the second eigenvector, the pixel point f ij The adjacent pixel points of the pixel point f correspond to the third eigenvector. ijThe corresponding relationship between the eigenvectors of the first eigenvector matrix and the eigenvectors of the first eigenvector matrix is ​​used to determine an eigenvector of the first eigenvector matrix corresponding to the eigenvector, as the second eigenvector. Similarly, the corresponding third eigenvector can be determined based on the corresponding relationship between each adjacent pixel point and the eigenvectors in the first eigenvector matrix. It can be understood that the pixel point f ij There is at least one adjacent pixel point, so there is at least one corresponding third eigenvector. The Euclidean distance between the second eigenvector and each third eigenvector can be calculated separately, and at least one Euclidean distance value can be obtained. If only one Euclidean distance value is obtained, the value of the Euclidean distance can be used as the significance value. If multiple Euclidean distance values ​​are obtained, the multiple Euclidean distance values ​​can be averaged and the averaged value can be used as the significance value. It can be understood that the farther the Euclidean distance between the second eigenvector and the third eigenvector, the larger the value of the Euclidean distance, and the larger the significance value, that is, the pixel point f ij The higher the significance.

[0061] In the above technical solution, the Euclidean distance between the second eigenvector and the third eigenvector is calculated as the significance value. The calculation method is simple and does not require complex calculations.

[0062] Exemplarily, the adjacent pixel points also include the image P to be processed in the K images to be processed. k The first pixel point is a pixel point adjacent to the second pixel point in the other image to be processed, and the second pixel point is a pixel point adjacent to the pixel point f ij In some embodiments, the sizes of the K images to be processed may be the same. The positions of the pixels in each image to be processed may correspond one to one. k The pixel point f ij The position of the image to be processed P is determined k The pixel point f in other images to be processed ij For each of the other images to be processed, the adjacent pixels of each second pixel can be determined, and the adjacent pixels of each second pixel can be the first pixel. k The pixel point f ij The adjacent pixel points of also include the first pixel point. In other words, the pixel point f ij The neighboring pixel points of the pixel point include the neighboring pixel points in the processed image, and also include the neighboring pixel points of the pixel point f in other images to be processed. ijThe adjacent pixels of the corresponding pixel. Thus, when calculating the significance value, the calculation can be performed based on the eigenvector corresponding to each adjacent pixel. In some embodiments, the significance value can be calculated based on the eigenvector in the first eigenvector matrix of the image to be processed corresponding to each adjacent pixel. In other embodiments, the significance value can be calculated based on a vector composed of multi-channel data in the image to be processed corresponding to each adjacent pixel. For example, the image to be processed may include three RGB channels, and the vector corresponding to each adjacent pixel can be determined based on the RGB value corresponding to each adjacent pixel.

[0063] In the above technical solution, the adjacent pixel points also include the pixel point f in other images to be processed. ij The corresponding pixel points are adjacent to the pixel points. Therefore, when calculating the saliency value, the features of all the images to be processed are included, which improves the robustness of the saliency value and further improves the accuracy of determining the defect location.

[0064] For example, for each image P to be processed k , the image to be processed P k The pixels in the image P are respectively k Step S1400 corresponds to the eigenvector in the first eigenvector matrix of the image to be processed P k The first eigenvector matrix and the image to be processed P k The defect position in the image to be processed is determined k The method comprises: step S1410 and step S1420.

[0065] In step S1410, in the image to be processed P k In the first eigenvector matrix, determine the image P to be processed k The feature vector corresponding to the pixel point at the defect position in the image P can be determined. k All the corresponding pixels within the defect position range can be defected to the pixel corresponding to the defect position. In other words, the defect position corresponds to multiple pixels. The eigenvector of the defect position pair can be determined according to the corresponding relationship between the pixel corresponding to the defect position and the eigenvector in the first eigenvector matrix.

[0066] In step S1420, the weighted average of the determined feature vectors is calculated to obtain the image P to be processed. k The first eigenvector of the image to be processed can be the weighted average of the eigenvectors of the defect position. It can be calculated according to the following formula:

[0067]

[0068] Among them, f all (c) represents the first eigenvector; f(i, j, c) represents the pixel point f at the defect location ij The eigenvector in the corresponding first eigenvector matrix; w(i, j) represents the pixel point f ij Corresponding weight; N is the pixel point f corresponding to the defect position ij In some embodiments, the weight of each pixel may be 1 / N. In other words, the first eigenvector may be the average of the eigenvectors in the first eigenvector matrix corresponding to the pixel at the defect position. In other embodiments, the weight of each pixel may be different. For example, the corresponding weight of the pixel may be determined according to its influence on the defect.

[0069] The above technical solution obtains the first eigenvector of the image to be processed by calculating the weighted average of the eigenvectors corresponding to the defect position. The calculation method is simple and efficient, and can better represent the characteristics of the defect position of the image to be processed to improve the accuracy of classification.

[0070] Exemplarily, step S1500 includes: step S1510 according to each to-be-processed image P k The first eigenvectors of each image are used to cluster all the images to be processed to obtain clustering results. Clustering is an unsupervised learning method. There is no need to collect sample data to train the model. Clustering models may include existing or future developed clustering models such as K-means and hierarchical clustering. Before all first eigenvectors are input into the clustering model, all first eigenvectors may be preprocessed. For example, all first eigenvectors may be standardized and normalized. All preprocessed first eigenvectors are input into the clustering model to obtain clustering results. The clustering results may include clustering clusters corresponding to each first eigenvector. The clustering clusters may be mapped to classification results. Specifically, the classification result corresponding to the classification cluster may be determined according to the defect form in the image to be processed corresponding to the first eigenvector in each clustering cluster. For example, if the defects in the image to be processed corresponding to a certain clustering cluster are all in the form of dirt, then the classification result of the image to be processed corresponding to the clustering cluster may be determined as dirt.

[0071] The above technical solution classifies the image to be processed by clustering the first feature vectors corresponding to the defect positions, thereby significantly improving the accuracy of image defect classification and enhancing the classification effect.

[0072] Exemplarily, step S1510 is based on each image P to be processed. k The respective first eigenvectors are used to cluster all the images to be processed to obtain a clustering result, including: step S1511 and step S1512.

[0073] In step S1511, all first eigenvectors are spliced ​​to obtain a second eigenvector matrix. The dimensions of all first eigenvectors may be the same. All first eigenvectors may be spliced ​​in the order of their corresponding images to be measured to obtain a second eigenvector matrix. In other words, the second eigenvector matrix includes all first eigenvectors.

[0074] In step S1512, all the images to be processed are clustered according to the second eigenvector matrix to obtain a clustering result. The second eigenvector matrix can be input into a clustering model, and each first eigenvector in the second eigenvector matrix is ​​clustered to obtain a clustering result to classify the images to be processed.

[0075] The above technical solution obtains the second eigenvector matrix by concatenating all the first eigenvectors, and clusters the second eigenvector matrix, thereby facilitating clustering calculation and improving the processing speed of the image processing method.

[0076] Exemplarily, the image processing method further includes: displaying the classification results of the K images to be processed. The classification results of the K images to be processed can be displayed by a visualization method. In some embodiments, visualization can be performed based on the first eigenvector of each image to be processed. For example, all first eigenvectors can be reduced in dimension so as to display the position of each first eigenvector in a two-dimensional coordinate system. The first eigenvectors belonging to the same classification result can be displayed in the same color, and different classification results can be displayed in different colors. Furthermore, the image to be processed corresponding to the classification result can also be displayed so as to intuitively display the classification category. Figure 4 FIG. 4 shows a classification result display diagram according to an embodiment of the present invention. Figure 4 As shown in the figure, there are four classification results, namely 0, 1, 2, and 3. The first eigenvector corresponding to each classification result is displayed in the form of dots in the figure, and the classification results are distinguished by different colors. Each classification result also displays the corresponding image to be processed, so that the defect category corresponding to the classification result can be intuitively determined.

[0077] The above technical solution displays the classification results of all images to be processed, and intuitively displays the classification results to improve user experience.

[0078] Illustratively, according to another aspect of the present invention, an image processing device is also provided. Figure 5 FIG. 5 shows a schematic block diagram of an image processing apparatus 500 according to an embodiment of the present invention. Figure 5 As shown, the image processing device 500 includes an image acquisition module 510 , a feature extraction module 520 , a first determination module 530 , a second determination module 540 and a classification module 550 .

[0079] The image acquisition module 510 is used to acquire K images of the target object to be processed, where K is a positive integer greater than 1. The feature extraction module 520 is used to extract the feature of each image P of the K images to be processed. k , extract the image to be processed P k The first determining module 530 is used to determine each image to be processed P k The second determination module 540 is used to determine the defect position of the defect of the target object in each image to be processed P k , according to the image to be processed P k The first eigenvector matrix and the image to be processed P k The defect position in the image to be processed is determined k The classification module 550 is used to classify each image P to be processed according to the k The respective first feature vectors are used to classify the K images to be processed, wherein the defect types of the target objects in the images to be processed of different categories are different.

[0080] The first determination module 530 includes a first acquisition submodule, a first calculation submodule and a first determination submodule. The first acquisition submodule is used to k , get the image P to be processed k Each pixel f ij The adjacent pixel points include the adjacent pixel points of the image to be processed P k In the pixel f ij Adjacent pixel points, where i is less than or equal to the image to be processed P k The width of j is a positive integer less than or equal to the width of the image to be processed P. k The first calculation submodule is used for the image P to be processed. k Each pixel point f ij , according to the pixel point f ij The adjacent pixel points of the pixel point are calculated ij The first determination submodule is used to determine the significance value of the image to be processed P k Each pixel f ij The significance value of the image to be processed P is determined k A defect position in the image processing apparatus is defined as follows: wherein the significance value of the pixel at the defect position is greater than the significance value of the pixel outside the defect position.

[0081] For example, for each image P to be processed k , the image to be processed P k The pixel point and the image to be processed Pk The first calculation submodule includes a first calculation unit and a second calculation unit. The first calculation unit is used to calculate the image P to be processed. k The second eigenvector and at least one third eigenvector in the first eigenvector matrix are used to calculate the pixel point f ij With the pixel f ij The cosine similarity between adjacent pixels of ij Corresponding to the second eigenvector, the pixel point f ij The adjacent pixel points of the pixel point f respectively correspond to the third feature vector. The second calculation unit is used to calculate the pixel point f according to the cosine similarity. ij The significance value of .

[0082] For example, for each image P to be processed k , the image to be processed P k The pixel point and the image to be processed P k The first calculation submodule includes three calculation units. The third calculation unit is used to calculate the image P to be processed. k The second eigenvector and at least one third eigenvector in the first eigenvector matrix are used to calculate the pixel point f ij With the pixel f ij The Euclidean distance between adjacent pixels of is used as the significance value, where the pixel point f ij Corresponding to the second eigenvector, the pixel point f ij The adjacent pixel points of correspond to the third eigenvector respectively.

[0083] Exemplarily, the adjacent pixel points also include the image P to be processed in the K images to be processed. k The first pixel point is a pixel point adjacent to a second pixel point in the other image to be processed, and the second pixel point is a pixel point adjacent to the pixel point f ij The pixel corresponding to the position.

[0084] For example, for each image P to be processed k , the image to be processed P k The pixel point and the image to be processed P k The second determination module 540 includes: a second determination submodule and a second calculation submodule. The second determination submodule is used to determine the feature vector in the first feature vector matrix of the image to be processed. k In the first eigenvector matrix, determine the image P to be processed kThe second calculation submodule is used to calculate the weighted average of the determined feature vectors to obtain the image P to be processed. k The first eigenvector of .

[0085] Exemplarily, the classification module 550 includes a clustering submodule. The clustering submodule is used to classify each image P to be processed. k The first eigenvectors of each are used to cluster all the images to be processed to obtain clustering results.

[0086] Exemplarily, the clustering submodule includes a splicing unit and a clustering unit. The splicing unit is used to splice all first eigenvectors to obtain a second eigenvector matrix. The clustering unit is used to cluster all images to be processed according to the second eigenvector matrix to obtain a clustering result.

[0087] Exemplarily, the image processing apparatus 500 further includes a display module. The display module is used to display the classification results of the K images to be processed.

[0088] Illustratively, according to yet another aspect of the present invention, an electronic device is also provided. Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. The electronic device 600 includes a processor 610 and a memory 620. The memory 620 stores computer program instructions, which are used by the processor 610 to execute the image processing method described above when executed.

[0089] Exemplarily, according to another aspect of the present invention, a storage medium is also provided, wherein program instructions are stored on the storage medium, and the program instructions are used to execute the image processing method as described above when running. The storage medium may include, for example, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media.

[0090] Illustratively, according to yet another aspect of the present invention, there is further provided a computer program product, comprising computer program instructions, which are used to execute the image processing method as described above when run.

[0091] A person skilled in the art may understand the specific implementation schemes and beneficial effects of the above-mentioned image processing apparatus, electronic device, storage medium and computer program product by reading the above-mentioned description of the image processing method, which will not be elaborated here for the sake of brevity.

[0092] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0093] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0095] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0096] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0097] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0098] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0099] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the image processing device according to the embodiment of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0100] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0101] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple images of the target object to be processed; For each image to be processed among the plurality of images to be processed, extracting a first eigenvector matrix of the image to be processed; Determining a defect position of a defect of the target object in each image to be processed; For each image to be processed, determining a first eigenvector of the image to be processed according to a first eigenvector matrix of the image to be processed and a defect position in the image to be processed; The multiple images to be processed are classified according to the first feature vector of each image to be processed, wherein the defect types of the target objects in the images to be processed of different categories are different.

2. The image processing method according to claim 1, characterized in that: The determining of the defect position of the defect of the target object in each image to be processed includes: For each image to be processed, Acquire neighboring pixel points of each pixel point in the image to be processed, wherein the neighboring pixel points include pixel points in the image to be processed that are adjacent to the pixel point; For each pixel in the image to be processed, calculating the significance value of the pixel according to the adjacent pixels of the pixel; The defect position in the image to be processed is determined according to the significance value of each pixel in the image to be processed, wherein the significance value of the pixel at the defect position is greater than the significance value of the pixel outside the defect position.

3. The image processing method according to claim 2, characterized in that: For each image to be processed, a pixel point in the image to be processed corresponds to an eigenvector in a first eigenvector matrix of the image to be processed; The calculating the significance value of the pixel point according to the adjacent pixels of the pixel point includes: Calculate the cosine similarity between the pixel point and its neighboring pixel points according to the second eigenvector and at least one third eigenvector in the first eigenvector matrix of the image to be processed, wherein the pixel point corresponds to the second eigenvector, and the neighboring pixel points of the pixel point correspond to the third eigenvectors respectively; According to the cosine similarity, the significance value of the pixel is calculated.

4. The image processing method according to claim 2, characterized in that: For each image to be processed, a pixel point in the image to be processed corresponds to an eigenvector in a first eigenvector matrix of the image to be processed; The calculating the significance value of the pixel point according to the adjacent pixels of the pixel point includes: According to the second eigenvector and at least one third eigenvector in the first eigenvector matrix of the image to be processed, the Euclidean distance between the pixel point and its adjacent pixel points is calculated as the significance value, wherein the pixel point corresponds to the second eigenvector, and the adjacent pixel points of the pixel point correspond to the third eigenvectors respectively.

5. The image processing method according to any one of claims 2 to 4, characterized in that: The adjacent pixel points also include a first pixel point in other images to be processed among the multiple images to be processed other than the image to be processed, the first pixel point is a pixel point adjacent to a second pixel point in the other images to be processed, and the second pixel point is a pixel point corresponding to the position of the pixel point.

6. The image processing method according to any one of claims 1 to 4, characterized in that: For each image to be processed, a pixel point in the image to be processed corresponds to an eigenvector in a first eigenvector matrix of the image to be processed; The step of determining the first eigenvector of the image to be processed according to the first eigenvector matrix of the image to be processed and the defect position in the image to be processed comprises: In a first eigenvector matrix of the image to be processed, determining an eigenvector corresponding to a pixel point at the defect position in the image to be processed; A weighted average of the determined eigenvectors is calculated to obtain a first eigenvector of the image to be processed.

7. The image processing method according to any one of claims 1 to 4, characterized in that: The classifying the plurality of images to be processed according to the first feature vector of each image to be processed comprises: All the images to be processed are clustered according to the first eigenvector of each image to be processed to obtain a clustering result.

8. The image processing method according to claim 7, characterized in that: The clustering of all the images to be processed according to the first feature vector of each image to be processed to obtain a clustering result includes: Concatenate all first eigenvectors to obtain a second eigenvector matrix; All the images to be processed are clustered according to the second eigenvector matrix to obtain a clustering result.

9. The image processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: The classification results of the multiple images to be processed are displayed.

10. An image processing device, characterized in that: include: An image acquisition module, used to acquire multiple images to be processed of the target object; A feature extraction module, configured to extract a first feature vector matrix of each image to be processed among the plurality of images to be processed; A first determination module is used to determine a defect position of a defect of the target object in each image to be processed; A second determination module is used to determine, for each image to be processed, a first eigenvector of the image to be processed according to a first eigenvector matrix of the image to be processed and a defect position in the image to be processed; The classification module is used to classify the multiple images to be processed according to the first feature vector of each image to be processed, wherein the defect types of the target objects in the images to be processed of different categories are different.

11. An electronic device, comprising: A processor and a memory, characterized in that The memory stores computer program instructions, which are used by the processor to execute the image processing method according to any one of claims 1 to 9 when executed.

12. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the image processing method according to any one of claims 1 to 9 when running.

13. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the image processing method according to any one of claims 1 to 9 when executed.

Images