Image processing method, image processing device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method, an image processing device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring a plurality of original background images and original defect images of a target object; performing feature extraction on the plurality of original background images and the original defect images to obtain a plurality of first feature vectors corresponding to the plurality of original background images and second feature vectors corresponding to the original defect images; determining at least one target background image in the plurality of original background images according to the first feature vector and the second feature vector; for the defect area of the target object in the original defect image, determining a defect foreground area map in the original defect image; and fusing the defect foreground region map into the target background image to obtain a target fused image. Effective image amplification is carried out based on the original defect image, so that the diversity, reliability and fidelity of the amplified image are ensured; the scheme can be automatically executed, manual participation is not needed, and manpower is saved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically to an image processing method, an image processing device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the improvement of computer processing power, computer vision technology is increasingly applied to various scenarios, such as the identification and classification of object defects in industrial inspection. In the image-based object defect detection technology, the appearance of the object can be detected by an artificial intelligence model to determine the defects in the object. When the artificial intelligence model detects the quality of the object by appearance, it needs to be trained using the corresponding image data. However, a notable feature of industrial inspection scenarios is that the number of defective objects is small and the number of normal objects is large. However, the artificial intelligence model used for defect detection requires a large number of images for training, so the training sample images need to be amplified.

[0003] In some related methods, traditional image processing methods are used to enhance or amplify the original image, such as flipping, rotating, randomly scaling, changing brightness contrast, etc. Although these methods can greatly amplify the training sample images, the amplified images are only simple transformations or enhancements of the original images, and the image diversity is insufficient, so the improvement on the model training results is limited. In other methods, image fusion methods are used to extract the defective areas in the image of the object with defective areas and fuse them into the manually specified background image. Although this type of method is simple, the realism of the fused image is low and it is difficult to meet the needs of the application scenario. Summary of the invention

[0004] The present invention has been made in view of the above-mentioned problems.

[0005] According to one aspect of the present invention, there is provided an image processing method, comprising:

[0006] Acquire multiple original background images and original defect images of the target object; perform feature extraction on the multiple original background images and the original defect images respectively to obtain multiple first feature vectors corresponding to the multiple original background images and second feature vectors corresponding to the original defect images; determine at least one target background image from the multiple original background images based on the first feature vectors and the second feature vectors; determine a defect foreground area map in the original defect image for the defect area of ​​the target object in the original defect image; and fuse the defect foreground area map into the target background image to obtain a target fused image.

[0007] This not only performs effective image amplification based on the original defect image, but also ensures the diversity, reliability and realism of the amplified image; in addition, the above scheme can be automatically executed without human participation, saving manpower.

[0008] Exemplarily, determining the defect foreground region map in the original defect image includes: cropping the original defect image for the defect region in the original defect image to obtain a defect region image; and performing image segmentation on the defect region image to obtain a defect foreground region map. The defect foreground region map can be obtained more accurately, improving the accuracy of the image processing method, and thus improving the quality of the target fused image.

[0009] Exemplarily, performing image segmentation on the defect area image also obtains a defect background area map in the defect area image; the image processing method further includes: determining a target area in the target background image that corresponds to the defect background area map; fusing the defect foreground area map into the target background image to obtain a target fused image, including: fusing the defect foreground area map into the target area to obtain a target fused image. In the target fused image, the defect area is located in a similar area to that in the original defect image, and thus the target fused image is more realistic and has a more ideal effect.

[0010] Exemplarily, determining the target area in the target background image that corresponds to the defect background area map includes: gridding the target background image to obtain multiple grid areas; calculating the first grayscale distribution histogram of all grid areas; calculating the second grayscale distribution histogram of the defect background area map; respectively determining the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of all grid areas; and selecting the target area from all grid areas based on the similarity, wherein the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of the target area is greater than the second similarity threshold. Not only does it save computing power, but the obtained target area is close to the defect background area map, which ensures the accuracy of the determined target area, and thus ensures the fidelity of the target fused image.

[0011] Exemplarily, gridding the target background image includes: gridding the target background image according to the size of the defect area image so that the size of all grid areas is the same as the size of the defect area image. The size of the grid area is the same as the defect area image, which can more accurately determine the target area and further improve the realistic effect of the target fused image.

[0012] Exemplarily, at least one target background image is determined from a plurality of original background images based on the first eigenvector and the second eigenvector, including: respectively calculating the vector distance between each of the plurality of first eigenvectors and the second eigenvector; and selecting the target background image from the plurality of original background images based on the vector distance, wherein the vector distance between the first eigenvector and the second eigenvector of the target background image is less than a first distance threshold. The similarity between the plurality of original background images and the original defect image is measured by the vector distance, the algorithm is simple, the calculation efficiency is high, and the time required for the image processing method is reduced.

[0013] Exemplarily, the vector distance is the Euclidean distance. The vector distance adopts the Euclidean distance metric, which can not only more accurately determine the similarity between the first eigenvector and the second eigenvector, and then more accurately select the target background image in the original background image, thereby ensuring the fidelity of the target fused image, but also has lower requirements on the computing power of the system.

[0014] Exemplarily, before fusing the defect foreground region map into the target background image, the image processing method further includes: performing image transformation on the defect foreground region map to obtain an updated defect foreground region map, which can improve the diversity of the defect foreground region map and thus improve the diversity of the target fused image.

[0015] According to another aspect of the present invention, an image processing device is also provided, including an acquisition module, an extraction module, a first determination module, a second determination module and a fusion module. The acquisition module is used to acquire multiple original background images and original defect images of the target object; the extraction module is used to perform feature extraction on the multiple original background images and the original defect images respectively to obtain multiple first feature vectors corresponding to the multiple original background images and second feature vectors corresponding to the original defect images; the first determination module is used to determine at least one target background image from the multiple original background images according to the first feature vector and the second feature vector; the second determination module is used to determine a defect foreground area map in the original defect image for the defect area of ​​the target object in the original defect image; and the fusion module is used to fuse the defect foreground area map into the target background image to obtain a target fused image.

[0016] According to another aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned image processing method when the processor is executed.

[0017] According to yet another aspect of the present invention, there is further provided a storage medium on which program instructions are stored. The program instructions are used to execute the image processing method as described above when running.

[0018] According to another aspect of the present invention, a computer program product is provided, comprising computer program instructions, which are used to execute the image processing method as described above when running.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention.

[0021] Figure 1 A schematic flow chart of an image processing method according to an embodiment of the present invention is shown;

[0022] Figure 2 A schematic diagram of determining a target background image according to an embodiment of the present invention is shown;

[0023] Figure 3A A schematic diagram showing a defect foreground region map in a defect region image according to an embodiment of the present invention is shown;

[0024] Figure 3B Shows Figure 3A A schematic diagram of a defect background region map in a defect region image of the illustrated embodiment;

[0025] Figure 4 shows a grid area of ​​a target background image according to an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of determining a target area according to an embodiment of the present invention is shown;

[0027] Figure 6 A schematic diagram showing an image processing method according to an embodiment of the present invention is shown;

[0028] Figure 7 A schematic block diagram of an image processing device according to an embodiment of the present invention is shown;

[0029] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more obvious, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

[0031] In order to at least solve the above technical problems, the present invention provides an image processing method. The image processing method can be applied to any application scenario where an image needs to be amplified. For example, in industrial production, products with defects often account for a small proportion, while a large number of products do not have defects. Therefore, the number of original images of defective products (referred to as original defect images) is small, and they need to be amplified. The number of original images of products without defects (referred to as original background images) is large. The original defect image can be amplified using the original background image. Specifically, based on the feature vectors of the original defect image and multiple original background images, an original background image similar to the original defect image can be determined as a target background image, and the defect foreground area map in the original defect image can be fused into the target background image to obtain a target fused image. Thus, multiple different original defect images can be fused with their similar target background images respectively. A large number of diverse and more realistic target fused images can be provided as training images for the training of artificial intelligence models for defect detection.

[0032] Figure 1 FIG. 1 is a schematic flow chart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110, S120, S130, S140 and S170.

[0033] In step S110 , a plurality of original background images and original defect images of the target object are acquired.

[0034] The target object may be any object. Specifically, it may include a wafer, a liquid crystal panel, etc. The background image may be obtained by photographing a target object with or without defects using a camera. The defect image may be obtained by photographing a target object with defects using a camera.

[0035] Exemplarily, a plurality of target objects may be photographed by a camera device to obtain a photographed image corresponding to each target object. Exemplarily, defects of the target objects in the photographed images may be manually identified. Alternatively, defects of the target objects in the photographed images may be identified by an artificial intelligence algorithm. The photographed image of the target object with defects is used as a defect image, and the photographed image of the target object without defects is used as a background image. The original background image may be all or part of the background image. A defect image may be selected from the plurality of defect images as the original defect image. In other words, each of the plurality of defect images may be used as an original defect image for performing the image processing method.

[0036] Optionally, in some specific embodiments, the acquired multiple original background images and original defect images may be images obtained by preprocessing the captured images, such as size standardization, grayscale conversion, histogram equalization, etc., to eliminate the influence of irrelevant factors on subsequent image processing.

[0037] In step S120, feature extraction is performed on the plurality of original background images and the original defect images respectively to obtain a plurality of first feature vectors corresponding to the plurality of original background images and a second feature vector corresponding to the original defect images.

[0038] In this step S120, feature extraction is performed on each of the multiple original background images and the original defect images to obtain respective feature vectors.

[0039] The first eigenvector is a set of values ​​that can characterize the key features of the original background image. The key features may be edges, textures, colors, etc. Similarly, the second eigenvector is a set of values ​​that can characterize the key features of the original defect image. Edge detection, spatial frequency domain (SIFT), directional gradient histogram (HOG) and other algorithms can be used to extract features from the original background image and the original defect image. It is also possible to use a pre-trained model to extract features from the original background image and the original defect image. For example, a convolutional neural network (CNN) model is used for feature extraction. The convolutional neural network model can be pre-trained based on public data or existing data. Each original background image corresponds to a first eigenvector. The original defect image corresponds to a second eigenvector. In order to ensure that the first eigenvector and the second eigenvector have the same dimension and extract the same type of key features, the same method can be used to extract features from the original background image and the original defect image.

[0040] After the first feature vector and the second feature vector are obtained, they can be stored in a database, and a corresponding relationship between each first feature vector and the original background image can be established. Similarly, a corresponding relationship between the second feature vector and the original defect image can be established.

[0041] Alternatively, there may be multiple defect images. In some specific embodiments, feature vectors corresponding to all defect images may be extracted. One of the defect images is selected as the original defect image, and the feature vector corresponding to the defect image is used as the second feature vector corresponding to the original defect image.

[0042] In step S130, at least one target background image is determined from a plurality of original background images according to the first eigenvector and the second eigenvector.

[0043] Exemplarily, the similarity of the second eigenvector and multiple first eigenvectors can be judged separately. Similarity refers to the degree of similarity between the second eigenvector and the first eigenvector. In some specific embodiments, the similarity between the second eigenvector and the first eigenvector can be measured by cosine similarity, vector distance, etc. For example, if the similarity between the second eigenvector and a certain first eigenvector is high, that is, the second eigenvector is more similar to the first eigenvector. It can be understood that the second eigenvector represents the characteristics of the original defect image, and the first eigenvector represents the characteristics of the original background image. If the second eigenvector is similar to a certain first eigenvector, the original defect image corresponding to the second eigenvector is similar to the original background image corresponding to the first eigenvector. The original background image corresponding to the second eigenvector can be determined as the target background image.

[0044] For example, Figure 2 FIG. 2 shows a schematic diagram of determining a target background image according to an embodiment of the present invention. Figure 2 As shown, the first feature vectors of multiple original background images are extracted to obtain a first feature vector library. The first feature vector library is used to store the first feature vector of each original defect image. The second feature vector of the original defect image is extracted. Similar images are retrieved in the first feature vector library according to the second feature vector. The original background image corresponding to the first feature vector that is more similar to the second feature vector is used as the target background image, and the target background image is stored in the target background image list to facilitate the subsequent acquisition of the fused image.

[0045] In step S140, for the defect area of ​​the target object in the original defect image, a defect foreground area map in the original defect image is determined.

[0046] The defect foreground area map refers to the part of the original defect image that is abnormal or does not meet the expected standards. The defect foreground area map can be an image of the defects of the target object. The defects corresponding to the defect foreground area map can be caused by errors, damage, surface flaws, foreign matter, etc. in the manufacturing process of the target object. The original defect image of the target object may include defective parts and parts other than defects, wherein the above-mentioned defect foreground area map can be obtained based on the defective parts. Compared with the non-defective area of ​​the original defect image, the defect foreground area map may have color differences, brightness differences, texture differences, etc. In some specific implementations, the defect foreground area map can be determined in the original defect image using threshold segmentation, feature matching methods, etc.

[0047] In step S170, the defect foreground area image is fused into the target background image to obtain a target fused image.

[0048] The target foreground area map is fused into the target background image. In some embodiments, the defect foreground area map can be pasted into the target background image to obtain a target fused image. In other embodiments, the defect foreground area map can be fused with the target background image using existing or future developed image fusion methods such as Poisson fusion, pyramid fusion, wavelet transform, and average method to obtain a target fused image.

[0049] Optionally, after the defect foreground area map is fused into the target background image, the fused image can be processed by color correction, brightness contrast adjustment, etc., so that the defect foreground area is better fused into the target background image, and the processed image is used as the target fused image.

[0050] In the above technical solution, the feature vectors of the original background image and the original defect image are extracted, the target background image is determined in the original background image according to the extracted feature vectors, and finally, the defect foreground area map in the original defect image is fused into the target background image to obtain the desired target fused image. This not only performs effective image amplification based on the original defect image, but also ensures the diversity, reliability and fidelity of the amplified image; in addition, the above solution can be automatically executed without manual participation, saving manpower.

[0051] Exemplarily, step S130 determines at least one target background image from a plurality of original background images according to the first eigenvector and the second eigenvector, and includes step S131 and step S132.

[0052] In step S131 , the vector distance between each of the plurality of first eigenvectors and the second eigenvector is calculated respectively.

[0053] Vector distance can measure the similarity between the first eigenvector and the second eigenvector. It can also measure the similarity between the original background image and the original defect image. The vector distance can be measured by Manhattan distance, Chebyshev distance, etc. The appropriate vector distance measurement method can be selected according to factors such as the type of target object, the dimension of the eigenvector, and the application scenario of image processing.

[0054] In step S132, a target background image is selected from a plurality of original background images according to the vector distance. Exemplarily, the vector distance between the first feature vector and the second feature vector of the target background image is less than a first distance threshold.

[0055] The first distance threshold may be a vector distance value. The first distance threshold may be automatically set. Alternatively, the first distance threshold may be manually set by a user. If different vector distance measurement methods are used, the first distance threshold may be changed according to the vector distance measurement methods. In other words, different vector measurement methods may correspond to different first distance thresholds.

[0056] If the vector distance between a first eigenvector and a second eigenvector is less than a first distance threshold, the first eigenvector and the second eigenvector are close in distance in the multidimensional space. In other words, the first eigenvector and the second eigenvector are similar. Therefore, the original background image corresponding to the first eigenvector and the original defect image corresponding to the second eigenvector are similar, and the original background image can be selected as the target background image.

[0057] In the above technical solution, the target background image is selected from multiple original background images by calculating the vector distance between the first eigenvector and the second vector. The similarity between multiple original background images and the original defect image is measured by the vector distance. The algorithm is simple and the calculation efficiency is high, which reduces the time required for the image processing method.

[0058] Exemplarily, the vector distance may be the Euclidean distance. The Euclidean distance is widely used in many fields such as image recognition, image classification, text analysis, and signal processing. The Euclidean distance refers to the true distance between the first eigenvector and the second eigenvector in n-dimensional space. n is the dimension of the first eigenvector and the second eigenvector. The vector distance between the first eigenvector and the second eigenvector may be calculated according to the following formula:

[0059]

[0060] Where a=(a1,a2,……a n ), which represents the first eigenvector; a1, a2, ..., a ，n Respectively represent the components of the first eigenvector in different dimensions; b=(b1,b2,……b n), which represents the second eigenvector; b1, b2, ..., b n They represent the components of the second eigenvector in different dimensions respectively; d represents the Euclidean distance.

[0061] In the above technical solution, the vector distance adopts the Euclidean distance metric, which can not only more accurately determine the similarity between the first eigenvector and the second eigenvector, and then more accurately select the target background image in the original background image, thereby ensuring the realism of the target fused image, but also has lower requirements on the computing power of the system.

[0062] Exemplarily, step S140 determines a defect foreground area map in the original defect image for the defect area of ​​the target object in the original defect image, and includes step S141 and step S142.

[0063] In step S141 , the original defect image is cropped for the defect area in the original defect image to obtain a defect area image.

[0064] Exemplarily, the original defect image can be compared with the template image of the target object to determine the defect area therein. The template image of the target object can be a standard image of the target object without any defects. Through the comparison operation, the position of the defect area in the original defect image can be roughly determined. The original defect image can be cropped according to the position of the defect area to obtain the defect area image. Specifically, the circumscribed polygon of the defect area, such as a circumscribed rectangle, can be obtained according to the shape and size of the defect area. According to the circumscribed polygon, the original defect image is cropped to obtain the defect area image. The image outside the circumscribed polygon can be cropped, and the image within the circumscribed polygon is retained to obtain the defect area image. Optionally, the circumscribed polygon can be appropriately expanded to obtain the defect area image. For example, for the circumscribed rectangle, a specific number of pixels can be expanded in its length direction and width direction to obtain the defect area image. The specific number of pixels can be determined according to the size of the original defect image, such as a value between 5-100.

[0065] Alternatively, an existing or future developed defect recognition method such as a threshold-based method or a feature-based method may be used to roughly determine the defect area in the original defect image, and then the original defect image is cropped based on the position of the defect area to obtain a defect area image.

[0066] In step S142, the defect area image is segmented to obtain a defect foreground area image.

[0067] As described above, the defect area image is obtained by cropping the original defect image, and its size is smaller than that of the original defect image. As mentioned above, the defect area image includes not only the actual part of the defect area but also the peripheral part of the defect area. In other words, the defect area image includes the defect area and the part outside the defect area, and the defect foreground area map can be an image composed of the defect area in the defect area image. In the above example, the defect area image can be a circumscribed rectangle. The shape of the defect foreground area map is the shape of the defect area, which is most likely not a figure with a regular shape.

[0068] In some embodiments, the defect area image may be segmented to obtain a defect foreground area map. Specifically, the defect area image may be segmented using any existing or future developed image segmentation algorithm to obtain a defect foreground area map. For the defect area image, the pixel value inside the defect area may be set to 1, and the pixel value outside the defect area outline may be set to 0 according to the segmentation result. Thus, a binary mask of the defect area image may be obtained. The defect area image may be cut according to the binary mask to obtain a defect foreground area map. Thus, the actual part of the defect area is cut out from the defect area image to obtain a defect foreground area map with a high probability of an irregular shape. Figure 3A FIG. 1 is a schematic diagram showing a defect foreground region map in a defect region image according to an embodiment of the present invention. Figure 3A As shown, Figure 3A The middle grey portion is the defect foreground region image 310 . Figure 3A The black part in the figure corresponds to the part outside the defect foreground area. Figure 3A In the illustrated embodiment, the defect region image is rectangular.

[0069] The above technical solution obtains the defect area image by cropping the original defect image, and then segments the defect area image to obtain the defect foreground area map. Thus, the defect foreground area map can be obtained more accurately, the accuracy of the image processing method can be improved, and the quality of the target fused image can be improved.

[0070] Exemplarily, step S142 performs image segmentation on the defect area image and also obtains a defect background area map in the defect area image. As described above, the defect area image includes a defect foreground area map and an area outside the defect foreground area map. The defect background area map may be the portion of the defect area image outside the defect foreground area map. The defect background area map does not include the actual area of ​​the defect. After the defect area image is segmented in step S142 and the defect foreground area map is obtained, the remaining portion of the defect area image is the defect background area map. Figure 3B Shows Figure 3A Schematic diagram of defect background area map in defect area image of the embodiment shown. Figure 3B As shown, Figure 3B The middle gray portion is the defect background area FIG320 . Figure 3B The black part corresponds to Figure 3A Defect foreground area diagram shown.

[0071] The image processing method may further include step S150, determining a target area in the target background image that corresponds to the defect background area map. The target area is an area in the target background image that is similar to the defect background area. In some specific embodiments, features of the defect background area map may be extracted. Based on the features of the defect background area map, an area in the target background image that is similar to the features of the defect background area map is matched as the target area. Alternatively, a template matching method may be used to find an area in the target background image that is similar to the defect background image as the target area.

[0072] Step S170 may include step S171, fusing the defect foreground area map to the target area to obtain a target fused image. As described above, in some embodiments, the defect foreground area map may be pasted into the target area of ​​the target background image to obtain a target fused image. In other embodiments, the defect foreground area map may be fused into the target area of ​​the target background image using existing or future developed image fusion methods such as Poisson fusion, pyramid fusion, wavelet transform, and average method to obtain a target fused image.

[0073] It can be understood that for different types of defects, the probability of their occurrence in different positions in the target object may be different. For example, for the target object is a wafer, its pin-marked defects usually appear on or around the pad. Different positions in the target object usually present similar image features. The above technical solution determines the target area in the target background image based on the defect background area map, and the characteristics of the target area are similar to the position area of ​​the defect area in the original defect image. The defect foreground area map is fused to the target area in the target background image to obtain a target fused image. As a result, in the target fused image, the defect area is located in a similar area to that in the original defect image, and thus, the target fused image is more realistic and the effect is more ideal.

[0074] Exemplarily, step S150 determines a target area in the target background image that corresponds to the defect background area map, and includes steps S151, S152, S153, S154, and S155.

[0075] In step S151, the target background image is gridded to obtain a plurality of grid areas.

[0076] Gridding refers to dividing the target background image into multiple regularly arranged grid areas. The shape of the grid area can be a geometric shape such as rectangle, triangle, pentagon, hexagon, etc. Figure 4 FIG. 2 shows a grid area of ​​a target background image according to an embodiment of the present invention. Figure 4 As shown, the target background image 410 is gridded into 16 grid areas 420, and the shape of the grid area 420 is rectangular. In some embodiments, the shape of the grid area can be the same as the shape of the defect area image.

[0077] In step S152, the first grayscale distribution histogram of each of all grid areas is calculated.

[0078] A grayscale histogram is a statistical chart that describes the frequency of occurrence of pixels of different gray levels in an image. A first grayscale histogram is a statistical chart that describes the frequency of occurrence of pixels of different gray levels in a grid area. The first grayscale distribution histogram can represent the distribution of pixels of each gray level in the grid area. If the grid area is not a grayscale image, the grid area can be converted into a grayscale image. For each grid area, each pixel can be traversed, the number of pixels of each gray level can be calculated, and the first grayscale distribution histogram of the grid area can be obtained.

[0079] In step S153, a second grayscale distribution histogram of the defect background area image is calculated.

[0080] The second grayscale distribution histogram is a statistical chart describing the frequency of occurrence of pixels of different grayscale levels in the defect background area image. The second grayscale distribution histogram can represent the distribution of pixels of each grayscale level in the grid area. If the defect background area image is not a grayscale image, the defect background area image can be converted into a grayscale image. Traverse each pixel in the defect background area image, calculate the number of pixels of each grayscale level, and obtain the second grayscale distribution histogram of the defect background area image.

[0081] In step S154, the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of all grid areas is determined respectively.

[0082] The similarity between the first grayscale distribution histogram and the second grayscale distribution histogram refers to the degree of closeness between the two in shape, distribution or pattern. The similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of each grid area can be determined by comparison criteria such as correlation coefficient, chi-square coefficient, intersection coefficient, and Buffett coefficient.

[0083] In step S155, a target area is selected from all grid areas according to the similarity, wherein the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of the target area is greater than a second similarity threshold.

[0084] The similarity of the corresponding images can be evaluated by the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram. Therefore. The higher the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram, the more similar the grid area corresponding to the first grayscale distribution histogram is to the defect background area corresponding to the second grayscale distribution histogram. The second similarity threshold can be set according to the similarity comparison standard. If the similarity between a first grayscale distribution histogram and a second grayscale distribution histogram is greater than the second similarity threshold, the grid area corresponding to the first grayscale distribution histogram is determined as the target area. In some embodiments, there may be a plurality of first grayscale distribution histograms and the second grayscale distribution histogram whose similarity is greater than the second similarity threshold, and the grid area corresponding to the first grayscale distribution histogram with the highest similarity can be selected as the target area. Alternatively, the similarity between the plurality of first grayscale distribution histograms and the second grayscale distribution histogram is greater than the second similarity threshold, and the grid areas corresponding to the plurality of grayscale distribution histograms can be used as the target area. Each target area of ​​the target background image corresponds to a defect foreground area map fused together.

[0085] The execution order of the above steps S151, S152 and S153 can be changed, for example, first execute step S153, and then execute step S151 and step S152 in sequence. It can be understood that step S152 calculates the first grayscale distribution histogram of each of all grid areas, and should be executed after step S151 grids the target background image to obtain multiple grid areas. Step S153 calculates the second grayscale distribution histogram of the defect background area image, and can be executed before step S154 determines the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of each of all grid areas.

[0086] For example, Figure 5 FIG. 4 is a schematic diagram showing a target area determined according to an embodiment of the present invention. Figure 5 As shown, the defect background region map can be obtained according to the original defect image. The target background image is gridded. The grayscale distribution histogram of each grid region is matched with the defect background region map, and the grid region with the highest matching degree is used as the target region of fusion.

[0087] The above technical solution obtains multiple grid areas by gridding the target background image. The target area is determined based on the similarity between the first grayscale distribution histogram of all grid areas and the second grayscale distribution histogram of the defect background area map. This technical solution not only saves computing power, but also the obtained target area is close to the defect background area map, which ensures the accuracy of the determined target area and further ensures the fidelity of the target fused image.

[0088] Exemplarily, step S151 grids the target background image, including step S151a, gridding the target background image according to the size of the defect area image so that the size of all grid areas is the same as the size of the defect area image. In some embodiments, the target background area may be gridded according to the shape and size of the defect area image. Thus, the size and shape of the grid area may be the same as the defect area image. For example, in the case where the defect area image is a rectangle, the length and width of the grid area in the target background image may be determined according to the length and width of the defect area image, respectively, so that the sizes of the two are the same.

[0089] The above technical solution meshes the target background according to the size of the defect area image. The size of the grid area is the same as the defect area image, which can more accurately determine the target area and further improve the realistic effect of the target fused image.

[0090] Exemplarily, before fusing the defect foreground area map into the target background image in step S170, the image processing method further includes: step S160 performing image transformation on the defect foreground area map to obtain an updated defect foreground area map.

[0091] For example, the shape and size of the defect foreground area image may be transformed, or image parameters such as brightness, contrast, and sharpness may be transformed. The defect foreground area image may also be transformed using existing or future research and development technologies such as image style transfer methods and image generation models. Multiple updated defect foreground area images may be obtained using different transformation methods. Each updated defect foreground area image may be fused with the target background image, thereby obtaining more target fused images.

[0092] The above technical solution obtains an updated defect foreground area map by performing image transformation on the defect foreground area map, thereby obtaining more target fused images, thereby improving the diversity of the defect foreground area map, thereby improving the diversity of the target fused images.

[0093] For example, Figure 6 FIG. 4 is a schematic diagram showing an image processing method according to an embodiment of the present invention. Figure 6 According to the method, similar image retrieval is performed in multiple original background images based on the original defect image to determine the target background image. There may be multiple target background images. The defect area image is determined based on the original defect image, and the binary mask corresponding to the defect area image is determined. Furthermore, the defect foreground area map and the defect background area map are determined based on the binary mask and the defect area image. For each target background image, similar area retrieval is performed based on the defect background area map, the target area is determined in the target background image, and the defect foreground area map is fused to the target area to obtain a target fused image.

[0094] Illustratively, according to yet another aspect of the present invention, an image processing device is also provided. Figure 7 FIG. 8 is a schematic block diagram of an image processing apparatus 700 according to an embodiment of the present invention. Figure 7 As shown, the image storage device 700 includes an acquisition module 710 , an extraction module 720 , a first determination module 730 , a second determination module 740 and a fusion module 750 .

[0095] The acquisition module 710 is used to acquire multiple original background images and original defect images of the target object. The extraction module 720 is used to perform feature extraction on the multiple original background images and the original defect images respectively to obtain multiple first feature vectors corresponding to the multiple original background images and second feature vectors corresponding to the original defect images. The first determination module 730 is used to determine at least one target background image from the multiple original background images based on the first feature vector and the second feature vector. The second determination module 740 is used to determine a defect foreground area map in the original defect image for the defect area of ​​the target object in the original defect image. The fusion module 750 is used to fuse the defect foreground area map into the target background image to obtain a target fused image.

[0096] Exemplarily, the second determination module 740 includes a cropping submodule and a segmentation submodule. The cropping submodule is used to crop the original defect image according to the defect area in the original defect image to obtain a defect area image. The segmentation submodule is used to perform image segmentation on the defect area image to obtain the defect foreground area image.

[0097] Exemplarily, the image segmentation of the defect area image also obtains a defect background area map in the defect area image. The image processing device 700 also includes a third determination module. The third determination module is used to determine the target area in the target background image that corresponds to the defect background area map. The fusion module 750 includes a first fusion submodule. The first fusion submodule is used to fuse the defect foreground area map to the target area to obtain the target fused image.

[0098] Exemplarily, the third determination module includes a gridding submodule, a first calculation submodule, a second calculation submodule, a first determination submodule and a first selection submodule. The gridding submodule is used to grid the target background image to obtain multiple grid areas. The first calculation submodule is used to calculate the first grayscale distribution histogram of each of all grid areas. The second calculation submodule is used to calculate the second grayscale distribution histogram of the defect background area image. The first determination submodule is used to respectively determine the similarity between the first grayscale distribution histogram of each of all grid areas and the second grayscale distribution histogram. The first selection submodule is used to select the target area from all grid areas based on the similarity, wherein the similarity between the first grayscale distribution histogram of the target area and the second grayscale distribution histogram is greater than a second similarity threshold.

[0099] Exemplarily, the gridding submodule includes a gridding unit, which is used to grid the target background image according to the size of the defect area image so that the size of all grid areas is the same as the size of the defect area image.

[0100] Exemplarily, the first determination module includes a third calculation submodule and a second selection submodule. The third calculation submodule is used to respectively calculate the vector distance between each of the plurality of first feature vectors and the second feature vector. The second selection submodule is used to select the target background image from the plurality of original background images according to the vector distance, wherein the vector distance between the first feature vector of the target background image and the second feature vector is less than a first distance threshold.

[0101] Exemplarily, before fusing the defect foreground region map into the target background image, the image processing device 700 includes a transformation module. The transformation module is used to perform image transformation on the defect foreground region map to obtain an updated defect foreground region map.

[0102] Illustratively, according to yet another aspect of the present invention, an electronic device is also provided. Figure 8 A schematic block diagram of an electronic device 800 according to an embodiment of the present invention is shown. The electronic device 800 includes a processor 810 and a memory 820. The memory 820 stores computer program instructions, which are used by the processor 810 to execute the image processing method described above when executed.

[0103] Exemplarily, according to another aspect of the present invention, a storage medium is also provided, wherein program instructions are stored on the storage medium, and the program instructions are used to execute the image processing method as described above when running. The storage medium may include, for example, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media.

[0104] Illustratively, according to yet another aspect of the present invention, there is further provided a computer program product, comprising computer program instructions, which are used to execute the image processing method as described above when run.

[0105] A person skilled in the art may understand the specific implementation schemes and beneficial effects of the above-mentioned image processing apparatus, electronic device, storage medium and computer program product by reading the above-mentioned description of the image processing method, which will not be elaborated here for the sake of brevity.

[0106] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0107] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0109] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0110] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0111] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0112] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0113] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the image processing device according to the embodiment of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0114] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0115] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple original background images and original defect images of the target object; Performing feature extraction on the multiple original background images and the original defect image respectively to obtain multiple first feature vectors corresponding to the multiple original background images and a second feature vector corresponding to the original defect image; Determine at least one target background image from the plurality of original background images according to the first feature vector and the second feature vector; For the defect area of ​​the target object in the original defect image, determine a defect foreground area map in the original defect image; as well as The defect foreground area map is fused into the target background image to obtain a target fused image.

2. The image processing method according to claim 1, characterized in that: The step of determining a defect foreground area map in the original defect image comprises: For the defect area in the original defect image, cropping the original defect image to obtain a defect area image; The defect area image is segmented to obtain the defect foreground area image.

3. The image processing method according to claim 2, characterized in that: The image segmentation of the defect area image also obtains a defect background area image in the defect area image; The image processing method further comprises: Determine a target area in the target background image that corresponds to the defect background area map; The step of fusing the defect foreground area map into the target background image to obtain a target fused image includes: The defect foreground area map is fused to the target area to obtain the target fused image.

4. The image processing method according to claim 3, characterized in that: The determining of the target area in the target background image corresponding to the defect background area map includes: Gridding the target background image to obtain a plurality of grid areas; Calculate the first grayscale distribution histogram of all grid areas; Calculating a second grayscale distribution histogram of the defect background area image; respectively determining the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of all the grid areas; According to the similarity, the target area is selected from among all the grid areas, wherein the similarity between the first grayscale distribution histogram and the second grayscale distribution histogram of the target area is greater than a second similarity threshold.

5. The image processing method according to claim 4, characterized in that: The gridding the target background image comprises: The target background image is gridded according to the size of the defect area image, so that the size of all the grid areas is the same as the size of the defect area image.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The step of determining at least one target background image from the plurality of original background images according to the first feature vector and the second feature vector comprises: respectively calculating a vector distance between each of the plurality of first eigenvectors and the second eigenvector; The target background image is selected from the multiple original background images according to the vector distance, wherein the vector distance between the first feature vector and the second feature vector of the target background image is less than a first distance threshold.

7. The image processing method according to claim 6, characterized in that: The vector distance is a Euclidean distance.

8. The image processing method according to claim 1, characterized in that: Before fusing the defect foreground area map into the target background image, the image processing method further includes: Performing image transformation on the defect foreground area map to obtain an updated defect foreground area map.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a plurality of original background images and original defect images of a target object; An extraction module, used for performing feature extraction on the multiple original background images and the original defect image respectively, so as to obtain multiple first feature vectors corresponding to the multiple original background images and a second feature vector corresponding to the original defect image; A first determining module, configured to determine at least one target background image from the plurality of original background images according to the first feature vector and the second feature vector; A second determination module is used to determine a defect foreground area map in the original defect image for the defect area of ​​the target object in the original defect image; as well as A fusion module is used to fuse the defect foreground area image into the target background image to obtain a target fused image.

10. An electronic device, comprising: A processor and a memory, characterized in that The memory stores computer program instructions, which are used by the processor to execute the image processing method according to any one of claims 1 to 8 when executed.

11. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the image processing method according to any one of claims 1 to 8 when running.

12. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the image processing method according to any one of claims 1 to 8 when executed.

Images