Defect detection method and apparatus, electronic device, storage medium and program product

Abstract

Disclosed in the embodiments of the present application are a defect detection method and apparatus, an electronic device, a storage medium and a program product. The method comprises: acquiring an image under test of a target workpiece and a template image of the target workpiece, and comparing the image under test with the template image so as to obtain a differential image; inputting the image under test and the differential image into a defect detection model, and executing the following processing: on the basis of the image under test and the differential image, extracting a target image feature and, on the basis of the target image feature, performing defect detection on the image under test to obtain defect detection box information of the image under test; positioning a defect area in the differential image and, according to the defect area, determining a first detection result of the target workpiece; and, according to the defect detection box information and the first detection result, determining a second detection result of the target workpiece.

Description

Defect detection methods, devices, electronic equipment, storage media and program products

[0001] This application claims priority to Chinese Patent Application No. 202311498351.5, filed on November 10, 2023, entitled "Defect Detection Method, Apparatus, Electronic Device and Storage Medium". Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a defect detection method, apparatus, electronic device, storage medium, and program product.

[0003] Background of the Invention

[0004] Currently, deep learning algorithms are commonly used to address product defect detection. However, the accuracy of defect detection using deep learning algorithms relies heavily on a large number of manually labeled defect samples. The more diverse the defect types or the lower the distinguishability of defect features, the larger the number of defect samples required, leading to high sample acquisition costs and low training efficiency. Furthermore, since deep learning algorithms generally employ supervised training, a lack of training samples can reduce the accuracy of defect detection.

[0005] Summary of the Invention

[0006] This application provides a defect detection method, apparatus, electronic device, storage medium, and program product that can improve the accuracy of defect detection while reducing dependence on training samples.

[0007] On one hand, embodiments of this application provide a defect detection method, including:

[0008] The test image of the target workpiece and the template image of the target workpiece are acquired, and the test image and the template image are compared to obtain a difference image;

[0009] The image to be tested and the difference image are input into the defect detection model, and the following processing is performed:

[0010] Based on the image to be tested and the difference image, the features of the target image are extracted;

[0011] Based on the features of the target image, defect detection is performed on the image to be tested to obtain defect detection box information in the image to be tested;

[0012] The defect region is located in the differential image, and a first detection result of the target workpiece is determined based on the defect region; and,

[0013] Based on the defect detection box information and the first detection result, a second detection result for the target workpiece is determined.

[0014] On the other hand, embodiments of this application also provide a defect detection device, including:

[0015] The first processing module is used to acquire the image to be tested of the target workpiece and the template image of the target workpiece, and compare the image to be tested with the template image to obtain a difference image;

[0016] The second processing module is used to input the image to be tested and the difference image into the defect detection model and perform the following processing: extracting target image features based on the image to be tested and the difference image; performing defect detection on the image to be tested based on the target image features to obtain defect detection box information in the image to be tested;

[0017] The third processing module is used to locate the defect region in the differential image and determine the first detection result of the target workpiece based on the defect region.

[0018] The fourth processing module is used to determine the second detection result of the target workpiece based on the defect detection box information and the first detection result.

[0019] On the other hand, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described defect detection method.

[0020] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described defect detection method.

[0021] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the defect detection method described above.

[0022] Brief description of the attached figures

[0023] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0024] Figure 1 is a schematic diagram of an optional implementation environment provided in an embodiment of this application;

[0025] Figure 2 is a schematic flowchart of an optional defect detection method provided in an embodiment of this application;

[0026] Figure 3 is a schematic diagram showing the comparison between the image to be tested and the template image provided in an embodiment of this application;

[0027] Figure 4 is a schematic diagram of the generation of defect detection box information of the image to be tested provided in an embodiment of this application;

[0028] Figure 5 is a schematic diagram of the preprocessing of the image to be tested and the difference image before feature extraction provided in the embodiment of this application;

[0029] Figure 6 is a schematic diagram of the first image feature extraction process provided in an embodiment of this application;

[0030] Figure 7 is a schematic diagram of the generation of convolutional branch data stream during the extraction of first image features according to an embodiment of this application;

[0031] Figure 8 is a schematic diagram of a first image feature extraction process provided in another embodiment of this application;

[0032] Figure 9 is a schematic diagram of the process of extracting first image features according to another embodiment of this application;

[0033] Figure 10 is a schematic diagram of extracting the second target convolutional features according to an embodiment of this application;

[0034] Figure 11 is a schematic diagram of the target image feature generation process provided in an embodiment of this application;

[0035] Figure 12 is a schematic diagram of the defect detection process provided in an embodiment of this application;

[0036] Figure 13 is a schematic diagram of the second detection result determination process provided in the embodiment of this application;

[0037] Figure 14 is a schematic diagram of the effect of morphological filtering provided in the embodiment of this application;

[0038] Figure 15 is a schematic diagram of the process for determining the correlation coefficient provided in an embodiment of this application;

[0039] Figure 16 is a schematic diagram of the similarity transformation correspondence between the template image and the image to be tested provided in the embodiments of this application;

[0040] Figure 17 is a schematic diagram of the training process of the defect detection model provided in the embodiment of this application;

[0041] Figure 18 is a schematic diagram of a defect detection process provided in another embodiment of this application;

[0042] Figure 19 is a schematic diagram of the defect detection box information generation process provided in another embodiment of this application;

[0043] Figure 20 is a schematic diagram of an optional overall process of the defect detection method provided in an embodiment of this application;

[0044] Figure 21 is a schematic diagram of an optional overall process of the defect detection method provided in an embodiment of this application;

[0045] Figure 22 is a schematic diagram of an optional structure of the defect detection device provided in an embodiment of this application;

[0046] Figure 23 is a partial structural block diagram of the terminal provided in an embodiment of this application;

[0047] Figure 24 is a partial structural block diagram of the server provided in an embodiment of this application.

[0048] Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that in various specific embodiments of this application, when processing data related to the characteristics of the target object, such as attribute information or a set of attribute information, is required, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this application need to obtain the attribute information of the target object, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the separate permission or consent of the target object will the necessary target object-related data necessary for the normal operation of embodiments of this application be obtained.

[0051] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0052] Defect detection typically refers to the detection of surface defects in an object. Machine vision technology is used to detect defects such as spots, pits, scratches, color differences, and missing parts on the surface of a workpiece, and defective areas can be marked.

[0053] The three primary colors (Red, Green, Blue, RGB) are a color model composed of the three primary colors of red, green, and blue. Various colors are obtained by changing the three color channels of red, green, and blue and superimposing them. In the fields of electronic displays, photography, and computer graphics, RGB is widely used to describe, control, and display color images, and is used as a format for saving and displaying photographs after they are captured by a camera.

[0054] Morphological image processing refers to a series of image processing techniques that process the shape features of images. The basic idea of ​​morphology is to use a special structuring element to measure or extract the corresponding shapes or features in the input image for further image analysis and target recognition.

[0055] Currently, deep learning algorithms are commonly used to address product defect detection. For example, in industrial manufacturing, artificial intelligence and deep learning technologies are often employed to inspect the appearance of products. When using deep learning algorithms for defect detection, supervised training of deep learning network models is typically used. Defects are labeled on images of the workpiece being inspected, and abnormal regions are identified using object detection or segmentation methods. Finally, defects are classified to achieve the desired defect detection effect.

[0056] However, manually labeling defects in images of quality inspection workpieces is costly, and the more types of defects there are or the lower the distinguishability of defect features, the larger the number of defect samples required. In addition, samples of some visually severe defects are difficult to obtain, resulting in deep learning algorithms not having learned the features of similar defects and being unable to accurately distinguish the defects of quality inspection workpieces. Therefore, the accuracy of defect detection using deep learning algorithms depends on a large number of manually labeled defect samples, but the difficulty and cost of obtaining samples reduce the accuracy of defect detection when samples are scarce.

[0057] To address the aforementioned issues, embodiments of this application provide a defect detection method, apparatus, electronic device, storage medium, and program product, which can improve the accuracy of defect detection while reducing dependence on training samples.

[0058] The methods provided in this application can be applied to different technical fields, including but not limited to artificial intelligence, cloud technology, computer vision technology, industrial automation and other scenarios.

[0059] Referring to Figure 1, which is a schematic diagram of an optional implementation environment provided by an embodiment of this application, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0060] Terminal 101 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, vehicle terminal, etc., but is not limited to these. Terminal 101 and server 102 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the application does not impose any limitations. Optionally, terminal 101 can acquire a test image of the target workpiece and then send the test image to server 102.

[0061] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0062] Additionally, server 102 can also be a node server in a blockchain network. Optionally, server 102 can pre-store defect detection models, and server 102 can also pre-store template images of target workpieces, or server 102 can receive template images of target workpieces sent by terminal 101.

[0063] When server 102 obtains the image to be tested and the template image of the target workpiece, it can compare the image to be tested with the template image to obtain a difference image. Then, the image to be tested and the difference image are respectively input into the pre-stored defect detection model to obtain the defect detection box information in the image to be tested. At the same time, the defect area can be located in the difference image. Based on the defect area, the first detection result of the target workpiece (i.e., a reference detection result) is determined. Then, based on the defect detection box information and the reference defect detection information, the second detection result of the target workpiece (i.e., the target detection result) is determined.

[0064] In this way, by combining differential contrast processing in deep learning algorithms, we can obtain features with high visual visibility through differential contrast, which can visually assist in defect detection, effectively improve the accuracy of defect detection, and reduce the dependence on training samples.

[0065] For example, terminal 101 can acquire the image to be tested and the template image of the target workpiece, and then send the image to be tested and the template image to server 102. After server 102 receives the image to be tested and the template image of the target workpiece, it can compare the image to be tested and the template image to obtain a difference image. The difference image can provide visual difference information of the target workpiece. Then, the image to be tested and the difference image are respectively input into the defect detection model to extract the target image features. By introducing difference information into the defect detection model, it helps to improve the accuracy of model prediction and improve the reliability of target image features.

[0066] Next, based on the features of the target image, defect detection is performed on the image under test to obtain the defect detection box information in the image under test. The difference image is used to highlight the features related to the defect region in the image under test, which helps to improve the defect detection model's ability to detect defects that lack training samples and have not been learned, and improves the accuracy of the defect detection box information.

[0067] Subsequently, the defect region is located in the differential image, and the first detection result of the target workpiece is determined based on the defect region. Then, the second detection result of the target workpiece is determined based on the first detection result and the defect detection box information. This allows for visual assistance in defect detection by utilizing the high visual visibility of the features obtained through comparison, effectively improving the accuracy of defect detection and reducing the dependence on training samples.

[0068] Referring to Figure 2, which is an optional flowchart of a defect detection method provided in an embodiment of this application, the defect detection method can be executed by a terminal, or by a server, or by a combination of a terminal and a server. In this embodiment, the method is described as being executed by a server. The defect detection method includes, but is not limited to, the following steps 201 to 204.

[0069] Step 201: Obtain the image to be tested and the template image of the target workpiece, and compare the image to be tested with the template image to obtain the difference image.

[0070] In one possible implementation, the image to be tested on the target workpiece can be acquired by a camera or sensor. The image to be tested can be a surface image, a cross-sectional image, an image obtained by infrared imaging, etc., depending on the object of defect detection and the detection requirements.

[0071] The template image of the target workpiece can be a reference image, that is, an image of a known normal or defect-free workpiece, used to compare with the image to be tested in order to detect differences or defects on the target workpiece. The template image can be obtained in advance during the design or manufacturing process of workpieces of the same category as the target workpiece.

[0072] Specifically, for surface defect detection, a camera can be used to acquire surface images of the target workpiece. The image to be tested can be an image of the front, side or other angles of the target workpiece, so as to comprehensively capture the features of the workpiece surface.

[0073] For internal defect detection or cross-sectional inspection, sensors (such as X-ray sensors or infrared sensors) can be used to acquire images of the internal structure or cross-section of the target workpiece as images to be tested.

[0074] Defect detection can be applied to a variety of scenarios, such as steel plate defect detection, floor defect detection, PCB board defect detection, film defect detection, LED bead defect detection, metal rod end face inspection, fabric wrinkle level assessment, etc. The target workpieces are steel plates, flooring, PCB boards, films, LED beads, metal rods, fabrics, etc.

[0075] In one possible implementation, during the defect detection process, the differences and changes between the image to be tested and the template image can be compared to generate a differential image, thereby visually identifying possible defects or anomalies on the target workpiece.

[0076] The color modes of the target workpiece's image to be tested and the template image can be the same or different. Specifically, if both the target workpiece's image to be tested and the template image can be color RGB images, then grayscale conversion is required before comparing the two images. If the target workpiece's image to be tested can be a color RGB image, while the template image can be a pre-stored grayscale binary image, then only grayscale conversion of the target image is required before comparing the two images.

[0077] Specifically, grayscale transformation processing can be performed on the test image and template image of the target workpiece, respectively. This includes: calculating the grayscale difference between the current pixel and its neighboring pixels in the image to obtain a pixel difference value; then comparing the pixel difference value with a preset grayscale difference threshold to obtain the binarized pixel value of the current pixel; and finally, by traversing each pixel in the image, obtaining the binarized pixel values ​​of all pixels. Neighboring pixels can refer to pixels surrounding each pixel in the image, all pixels in the image, or pixels at different scales corresponding to the current pixel's position.

[0078] The test image and template image are converted into binary images, specifically grayscale images. The grayscale level reflects the brightness value of each pixel in the image; a higher grayscale level indicates greater pixel brightness. After obtaining the grayscale images corresponding to the test image and template image, they can be compared. First, image matching is performed on both images. Morphological processing such as rotation, translation, and stretching is applied to the template image to align the target workpiece displayed in the template image with the target workpiece displayed in the test image. Then, image comparison processing is performed between the test image and the adjusted template image. This involves subtracting the corresponding pixel values ​​of the two images to reduce similarities and highlight the differences between them.

[0079] As shown in Figure 3, Figure 3 is a schematic diagram of comparing the image to be tested and the template image provided in the embodiment of this application. The absolute value obtained by subtracting the pixel value of each pixel in the image to be tested from the pixel value of the corresponding pixel in the adjusted template image is used as the pixel value of each pixel in the difference image. The specific calculation formula is as follows: Df(x,y)=abs[I(x,y)-T(x,y)] (1)

[0080] Where Df(x,y) represents the pixel value of the pixel at coordinates (x,y) in the difference image, I(x,y) represents the pixel value of the pixel at coordinates (x,y) in the image to be tested, and T(x,y) represents the pixel value of the pixel at coordinates (x,y) in the adjusted template image.

[0081] As shown in Figure 3, the differential image displays differential region A, indicating a difference between the image under test and the adjusted template image. By comparing the adjusted template image and the image under test, it can be found that the adjusted template image displays a structural region corresponding to differential region A, while the image under test does not display a structural region corresponding to differential region A. That is, the reference workpiece of the same category as the target workpiece has a first structure at the corresponding position in differential region A, while the target workpiece lacks a first structure at the corresponding position in differential region A.

[0082] In one possible implementation, the template image can be pre-stored on a server or terminal. After acquiring the test image of the target workpiece, target detection is performed on the target workpiece in the test image to determine the workpiece category corresponding to the target workpiece. Then, a template image of the same workpiece category is obtained based on the corresponding workpiece category and used as the template image of the target workpiece.

[0083] Step 202: Input the image to be tested and the difference image into the defect detection model and perform the following processing: extract the target image features based on the image to be tested and the difference image; perform defect detection on the image to be tested based on the target image features to obtain the defect detection box information in the image to be tested.

[0084] In one possible implementation, the defect detection model can be a neural network model built on deep learning algorithms for automatically detecting and identifying workpiece defects. The defect detection model can learn and extract features from input sample images and predict and classify defects based on the extracted features.

[0085] The defect detection model is trained using supervised training, which requires a large amount of labeled data. Furthermore, the model's performance and accuracy depend heavily on the quality and quantity of the training data, and it struggles to detect unlearned defect categories. Therefore, in addition to inputting the test image into the defect detection model for defect detection, a difference image is also input. This allows the model to extract target image features by combining the test image and the difference image.

[0086] Since the difference image is obtained by comparing the image to be tested with the template image, it contains the difference information between the two. Therefore, by introducing the difference image to provide image difference information for the defect detection model, the defect detection model can be assisted in extracting features from the image to be tested. Based on the image difference information, the features of the corresponding regions in the image to be tested can be extracted more accurately, which helps to improve the accuracy of subsequent defect detection.

[0087] In one possible implementation, the defect detection bounding box can be a rectangular box used to label and locate the defect region in the image to be tested, and the defect detection bounding box information can refer to the position coordinates of the defect detection bounding box in the image to be tested, and can also include the confidence of the defect detection model for the output of the defect detection bounding box.

[0088] Since the test image may not clearly reflect defects, but the difference image has high visual visibility, it can highlight visually obvious defects in the test image. The resulting target image features can help increase the sensitivity of the defect detection model to visually apparent defects. Furthermore, the defect detection model is trained using a supervised training method, and the target image features can supplement the difference information between the test image and the template image, enabling the defect detection model to have better detection capabilities for unlearned defects, thereby improving the generalization ability of the defect detection model.

[0089] Therefore, defect detection of the test image based on the features of the target image enables the defect detection model to identify defects that are very small, visually inconspicuous, or similar to the background, thereby improving detection accuracy. This can help reduce the false detection rate for normal workpieces and the false negative rate for defective workpieces. It can also help the defect detection model identify defect features that have not been learned in the test image, thereby improving the accuracy of generating defect detection boxes and thus improving the reliability of defect detection box information.

[0090] In one possible implementation, during the process of inputting the image to be tested and the difference image into the defect detection model to extract the target image features, a first image feature can be extracted from the image to be tested; a second image feature can be extracted from the difference image; and the first image feature and the second image feature can be fused to obtain the target image feature.

[0091] In one possible implementation, feature fusion can include weighted fusion, feature concatenation, feature overlay, feature selection, and other methods.

[0092] Specifically, a first weight can be assigned to the first image feature, and a second weight can be assigned to the second image feature. Then, the first image feature and the second image feature are weighted and summed to obtain the target image feature. The selection of the first weight and the second weight can be determined based on the importance of the feature, or it can be obtained through model training.

[0093] Specifically, the first image features and the second image features can be concatenated in a preset order to form a larger vector, which can conveniently integrate the information of the two features, expand the dimension of the feature vector, and enable the defect detection model to simultaneously utilize the features of the image under test and the features of the difference image, thereby improving the performance of the defect detection model.

[0094] Specifically, the first image features and the second image features can be superimposed using addition, subtraction, multiplication, convolution, or pooling operations to obtain the target image features.

[0095] Alternatively, by performing region filtering on the first image features and the second image features, some key feature information from the first image features and the second image features can be selected and recombined to form target image features.

[0096] The first image feature of the image under test captures the original information of the target workpiece, while the second image feature of the differential image captures the difference information between the target workpiece and the reference workpiece (the workpiece shown in the template image of the target workpiece), that is, the defect information that the target workpiece may have. By fusing the first image feature and the second image feature, the information of the target workpiece can be represented more comprehensively and accurately, which helps to improve the accuracy of defect identification in subsequent defect detection steps.

[0097] Referring to Figure 4, which is a schematic diagram of the generation of defect detection box information for the image under test provided in an embodiment of this application, firstly, the image under test (image A) and the differential image (differential image B) of the target workpiece are acquired. Then, the feature extraction module in the defect detection model is used to extract features from image A and differential image B respectively, thereby obtaining image feature A (i.e., the first image feature) of image A and differential image feature B (i.e., the second image feature) of differential image B. Next, image feature A and image feature B are fused to obtain target image feature C. Then, defect detection can be performed on image A based on target image feature C to obtain the defect detection box information of image A.

[0098] Introducing differential image feature B (i.e., the second image feature of the differential image) allows the defect detection model to focus more on the differences between the target workpiece and the reference workpiece, making defects that might otherwise be hidden in background noise or surface texture easier to detect, thus improving the sensitivity of the defect detection model to defects and consequently improving the accuracy of the defect detection box information.

[0099] Furthermore, the feature extraction modules used to extract features from image A and the difference image B can be different, and the two feature extraction modules can be trained separately using corresponding training samples. For example, if image A is a color image, the feature extraction module used to extract features from the image to be tested (image A) can be trained using the color image as a training sample. Conversely, if the difference image B is a grayscale image, the feature extraction module used to extract features from the difference image B can be trained using the grayscale image as a training sample. This allows for the extraction of more accurate image features for different image data.

[0100] In addition, the parameters of the feature extraction module used to extract features from both image A and the difference image B can be the same, that is, they can share the same feature extraction module. This allows the feature extraction module of the defect detection model to learn two different types of information at the same time and capture the difference information between the two images more accurately.

[0101] Step 203: Locate the defect area in the differential image and determine the first detection result of the target workpiece based on the defect area.

[0102] In one possible implementation, the pixel values ​​of each pixel in the difference image can be compared with a preset pixel threshold. When the pixel value of a pixel exceeds the preset pixel threshold, it can be considered that a defect exists at the location corresponding to that pixel. By counting the pixels whose pixel values ​​exceed the preset pixel threshold, the defect area can be located in the difference image.

[0103] In one possible implementation, the difference image can be a grayscale image. Connectivity analysis can be performed on the difference image to group adjacent pixels with the same pixel value into a connected component. Adjacent pixels can be foreground pixels, meaning pixels with an adjacent pixel value of 1 are grouped into a connected component. Then, the location and size information of the defect region are determined by calculating the bounding rectangle or minimum bounding circle of the connected component. For example, the bounding rectangle of the connected component can be calculated and used as the defect region.

[0104] In one possible implementation, the shape features of the connected regions (such as aspect ratio, roundness, etc.) can be further calculated to check whether the defect region conforms to the defect features, so as to filter out the defect regions that conform to the defect features and determine the first detection result.

[0105] In addition, before performing connected component analysis on the difference image, morphological processing (such as opening or closing operations) can be performed on the difference image to reduce the influence of noise. Connected components can also be filtered based on a preset connected component threshold (such as connected component area or connected component perimeter) to exclude connected components with too few pixels, thereby improving the accuracy of the analysis.

[0106] In one possible implementation, after the defect area is determined, the first detection result of the target workpiece can be determined by judging whether the defect area meets the defect characteristic conditions. That is, the first detection result is used to indicate whether the target workpiece has a defect based on the differential image.

[0107] Specifically, the defect feature conditions can be: the area, perimeter, shape of the defect region, or the difference between the pixel value of a pixel in a connected region and the pixel value of a pixel outside the connected region, and whether these exceed a preset defect threshold. If the defect region meets one or more defect feature conditions, the target workpiece can be considered to have a defect, i.e., the first inspection result is quality inspection failure; if the defect region does not meet the defect feature conditions, the target workpiece can be considered to have no defect, i.e., the first inspection result is quality inspection pass.

[0108] Step 204: Determine the second inspection result of the target workpiece based on the defect detection box information and the first inspection result.

[0109] In one possible implementation, the defect detection box information includes regions in the test image that may contain defects, predicted by the defect detection model based on features learned during training. The first detection result is used to indicate the defect status of the target workpiece obtained after comparing the template image and the test image. It has high visual visibility, can avoid missing visually obvious defects that did not appear during training, and has a high recall rate for known defects, thus effectively improving the detection accuracy.

[0110] Furthermore, computer vision technology is susceptible to interference from factors such as lighting conditions and workpiece surface texture, and some defects may be difficult to detect from a single viewpoint or dimension. However, combining multiple information sources can reduce the impact of interference factors, improve the stability and reliability of detection, and thus enable more comprehensive and accurate defect detection. Therefore, by combining defect detection box information and the first detection result to determine the second detection result, false positives and false negatives can be effectively reduced, thereby improving the accuracy of target defect detection while reducing dependence on training samples.

[0111] In one possible implementation, the quality inspection pass rate of the target workpiece can be determined by comparing the defect detection box information and the preset defect area conditions. For example, the preset defect conditions could be that the area of ​​the defect detection box indicated by the defect detection box information is larger than the preset defect area, or that the defect detection box indicated by the defect detection box information contains a non-defectable area (such as a structural area that seriously affects the quality of the workpiece). Therefore, when the defect detection box information or the first inspection result indicates that the target workpiece fails the quality inspection, it can be determined that the defect in the target workpiece cannot pass the quality inspection, i.e., the second inspection result is a quality inspection failure; when both the defect detection box information and the first inspection result indicate that the target workpiece passes the quality inspection, it can be determined that the target workpiece has no defects or that the existing defects can pass the quality inspection, i.e., the second inspection result is a quality inspection pass.

[0112] In one possible implementation, during the process of inputting the image to be tested and the difference image into the defect detection model, the data of each channel in the difference image can be copied until the number of channels in the copied difference image is equal to the number of channels in the image to be tested. The image to be tested and the copied difference image are then input into the defect detection model, wherein the image to be tested and the copied difference image share the same feature extraction module in the defect detection model.

[0113] Both the image to be tested and the difference image contain key information that helps identify defects. In order to enable the defect detection model to acquire more information and make more accurate predictions, both the image to be tested and the difference image can be input into the same feature extraction module for feature extraction. However, the feature extraction module requires all inputs to have the same dimension.

[0114] Referring to Figure 5, which is a schematic diagram of preprocessing before feature extraction of the image to be tested and the difference image provided in this embodiment of the application. In this case, taking the difference image (Image 1) as a 3x3 grayscale image as an example, Image 1 is a single-channel image, represented by matrix H. Each element in matrix H corresponds to a pixel in the image, and each element represents the brightness. The value range of each element can be from 0 to 255.

[0115] The image under test is a color image with multi-channel three-dimensional data. For example, the image under test (image 2) is a three-channel 3x3 color RGB image. In the matrix, the data of channel R represents the red channel component, the data of channel G represents the green channel component, and the data of channel B represents the blue channel component.

[0116] Therefore, the number of channels in the differential image can be made equal to the number of channels in the image under test by copying. As shown in Figure 5, the differential image (Image 1) is copied into an image with three channels (Image 3), where the data of channels H1 and H2 are the same as those of channel H.

[0117] The difference image and the image under test share the same feature extraction module in the defect detection model, which enables the defect detection model to process and learn information in two different representations in the same parameter space. This implicitly provides the model with difference information about the target workpiece, further improving the performance of the defect detection model. At the same time, since the feature extraction module reuses the same set of neural network weights, it does not need to process and learn a set of weights separately for each input, which can reduce the dependence on training samples and improve training efficiency.

[0118] In one possible implementation, during the extraction of the first image features of the image to be tested, multiple stages of convolution can be performed sequentially on the image to be tested, wherein...

[0119] Each stage includes at least one convolutional branch, and each convolutional branch is used to perform multiple convolutions;

[0120] When moving from the current stage to the next stage, a new convolutional branch is added, and the resolution of the new convolutional branch is less than that of any existing convolutional branch in the current stage.

[0121] The feature input to any convolutional branch in the current stage is obtained by fusing the features output by all convolutional branches in the previous stage;

[0122] Then, the initial convolutional features output from each convolutional branch in the last stage are fused together to obtain the first target convolutional feature;

[0123] Next, the first target convolutional features are downsampled to obtain the first image features.

[0124] First, the image under test is convolved through multiple stages. Each stage involves multiple convolutions, with the initial convolution only extracting low-level features such as lines or edges of the target object (e.g., the structure of the target workpiece). As the number of convolution stages increases, deeper features are effectively extracted, such as the texture and shape of the target object. This helps the defect detection model gain a deeper understanding of the specific content displayed in the image, enabling more accurate missing part detection. Furthermore, the multiple convolutions effectively reduce the image's spatial size, decreasing the amount of data required for subsequent computations.

[0125] After obtaining high-level image features, new convolutional branches can be added. The resolution of the added convolutional branches is lower than that of any existing convolutional branch in the same stage. The input of any existing convolutional branch in the same stage is obtained by fusing the outputs of all convolutional branches in the previous stage.

[0126] By adding convolutional branches with lower resolution, we can detect more globally significant or abstract features on the already extracted features, gradually capturing higher-level semantic information. At the same time, we can reduce noise in the features in the new convolutional branches and improve the reliability of the features.

[0127] Specifically, referring to Figure 6, which is a schematic diagram of the process for extracting the first image features provided in an embodiment of this application, the process of extracting the first image features of the image to be tested adopts a parallel convolutional stream approach and has multiple stages. Starting from the first stage, data streams are gradually added from high resolution to low resolution. The resolution of the parallel data stream in the later stage consists of the resolution of the previous stage and a lower resolution. As the number of stages increases, the number of different resolutions also increases.

[0128] As shown in Figure 6, the new convolutional branch is the data stream with the lowest resolution in the current stage. Its input is obtained by downsampling the features output by all convolutional branches in the previous stage. When there are multiple convolutional branches in the previous stage, the features output by all convolutional branches can be downsampled to obtain features with the same resolution as the convolutional branch to be added. Then, these features with the same resolution are fused (such as weighted fusion, concatenation, superposition, etc.) and used as the input of the new convolutional branch, so as to introduce a convolutional branch with a lower resolution and obtain image features with a lower resolution.

[0129] Referring to Figure 7, which is a schematic diagram of generating convolutional branch data streams during the extraction of first image features according to an embodiment of this application, the extraction process shown in Figure 7 corresponds to the extraction process shown in Figure 6. The process of extracting the first image features of the image to be tested can be divided into four stages, wherein:

[0130] The first phase consists of a data stream N corresponding to a resolution. 11 This resolution is the resolution of the original convolution branch that performs convolution on the image under test (the first resolution).

[0131] The second stage consists of two data streams corresponding to the two resolutions, including a data stream N for the first resolution. 21 and a data stream N at a resolution lower than the first resolution (second resolution) 22 The second resolution is smaller than the first resolution.

[0132] Similarly, the third stage contains three data streams N corresponding to three resolutions. 31 N 32 N 33 The fourth stage contains four data streams N corresponding to four resolutions. 41 N 42 N 43 N 44 This allows for the introduction of lower-resolution convolutional branches to obtain lower-resolution image features.

[0133] When adding new convolutional branches in a new convolutional stage, the features output by each existing convolutional branch in the previous stage are upsampled or downsampled accordingly and added to the input of the current convolution of the corresponding convolutional branch. This can preserve the original feature learning path of the previous stage and allow features at different resolution levels to be fused and exchange feature information at different resolutions.

[0134] Specifically, referring to Figure 8, which is a schematic diagram of a first image feature extraction process provided in another embodiment of this application. If the resolution of the current convolutional branch to be sampled is less than the resolution of the target convolutional branch, the output of the current convolutional branch to be sampled in the previous stage is upsampled to obtain an upsampled feature with the same resolution as the target convolutional branch. Then, the upsampled feature is used as the input of the target convolutional branch in the current stage, so that the target convolutional branch performs convolution on the upsampled feature.

[0135] If the resolution of the current convolutional branch to be sampled is greater than the resolution of the target convolutional branch, then the output of the current convolutional branch in the previous stage is downsampled to obtain a downsampled feature with the same resolution as the target convolutional branch. Then, this downsampled feature is used as the input of the target convolutional branch in the current stage, so that the target convolutional branch performs convolution on the downsampled feature.

[0136] In cases where multiple downsampling is required, multiple downsampling steps can be used to sample to the desired resolution.

[0137] If the resolution of the current convolutional branch to be sampled is equal to the resolution of the target convolutional branch, then the output of the current convolutional branch to be sampled in the previous stage is retained and used as the input of the convolutional branch (target convolutional branch) in the current stage, or the output of the current convolutional branch to be sampled in the previous stage is resampled and used as the input of the convolutional branch (target convolutional branch) in the current stage.

[0138] As shown in Figure 8, the first resolution D1 is greater than the second resolution D2, and the second resolution D2 is greater than the third resolution D3.

[0139] If the resolution of the target convolutional branch is the first resolution D1, then the convolutional branch L1 corresponding to the first resolution can directly add the convolutional features output in the previous stage to the input of the current stage. However, the convolutional branch L2 corresponding to the second resolution D2 and the convolutional branch L3 corresponding to the third resolution need to upsample the convolutional features output in the previous stage to the first resolution D1 and add them to the input of the convolutional branch L1 in the current stage.

[0140] If the resolution of the target convolutional branch is the second resolution D2, then the convolutional branch L1 corresponding to the first resolution needs to downsample its own convolutional features output in the previous stage to the second resolution D2 and add them to the input of the convolutional branch L2 in the current stage. The convolutional branch L2 corresponding to the second resolution can directly add its own convolutional features output in the previous stage to the input of the current stage. The convolutional branch L3 corresponding to the third resolution needs to upsample its own convolutional features output in the previous stage to the second resolution D2 and add them to the input of the convolutional branch L2 in the current stage.

[0141] If the resolution of the target convolutional branch is the third resolution D3, then the convolutional branch L1 corresponding to the first resolution and the convolutional branch L2 corresponding to the second resolution D2 need to downsample the convolutional features output in the previous stage to the third resolution D3 and add them to the input of the convolutional branch L3 in the current stage. The convolutional branch L3 corresponding to the third resolution can directly add the convolutional features output in the previous stage to the input of the current stage.

[0142] Therefore, when each stage begins convolution, the data input to each convolution branch is first fused to reduce the amount of data processing.

[0143] By repeatedly exchanging information in parallel data streams to perform multi-resolution fusion, high-resolution and low-resolution features are connected in parallel and advanced synchronously. The high-resolution features and low-resolution features continuously exchange information, and the extracted high-resolution features can make the first image features more spatially accurate in subsequent feature fusion steps, while the extracted low-resolution features can make the first image features more semantically complete in subsequent feature fusion steps.

[0144] In one possible implementation, in the process of fusing the initial convolutional features output from each convolutional branch in the last stage to obtain the first target convolutional feature, multiple initial convolutional features can be fused first to obtain fused features; for each convolutional branch in the last stage, the fused feature is subjected to a convolution with the same resolution as that convolutional branch to obtain the second target convolutional feature of that convolutional branch; multiple second target convolutional features are fused to obtain the first target convolutional feature.

[0145] Multiple second-target convolutional features with different resolutions are fused to obtain first-target convolutional features, so that the first-target convolutional features can contain more comprehensive and richer feature information.

[0146] Next, the first target convolutional features are downsampled and further compressed to obtain feature representations. More abstract and globally semantic features are extracted to obtain the first image features of the image to be tested.

[0147] Specifically, referring to Figure 9, which is a schematic diagram of the process of extracting the first image features provided in another embodiment of this application, as shown in Figure 9, after obtaining the initial convolutional features output by all convolutional branches (including convolutional branches L1, L2, L3 and L4) in the last stage, the initial convolutional features output by each convolutional branch are convolved based on the resolution of the convolutional branch itself, so that each convolutional branch can obtain the second target convolutional features with the same resolution as itself.

[0148] Next, all the second target convolutional features can be concatenated to form the first target convolutional feature. At this time, the number of channels of the first target convolutional feature is much larger than the number of channels of the image under test.

[0149] Then, the first target convolutional feature is downsampled, the target convolutional feature is compressed, and the dimension of the first target convolutional feature is reduced so that the resolution of the first image feature is equal to the minimum resolution of the output feature in all convolutional branches.

[0150] In one possible implementation, during the process of performing multiple stages of convolution on the image under test, when the minimum resolution of the current stage meets a preset resolution threshold, the addition of new convolution branches is stopped, and the current stage is taken as the last stage of the convolution process, that is, each convolution branch performs the last convolution.

[0151] Since introducing a new convolution branch increases the number of convolution operations, it can easily lead to an overly complex model and reduce training efficiency. In addition, too low a resolution may result in the loss of a lot of useful details, affecting the quality of the first image features obtained by subsequent feature fusion. Therefore, by judging whether the minimum resolution of the current stage meets the preset resolution threshold, it is determined whether the current stage should be the last stage of the convolution process.

[0152] When the current stage is the last stage, the output (i.e. the initial convolutional features) of the current convolutional branch in the last stage is upsampled or downsampled according to the resolution relationship between the current convolutional branch and the other convolutional branches, so as to obtain the convolutional features corresponding to the resolution of the other convolutional branches, and thus obtain the second target convolutional features of each convolutional branch.

[0153] Referring to Figure 10, which is a schematic diagram of extracting the second target convolutional features according to an embodiment of this application, as shown in Figure 10, when the resolution corresponding to the new convolutional branch (convolutional branch X) introduced in the Mth stage meets the preset resolution threshold, the Mth stage can be determined as the last stage of feature extraction. Therefore, the initial convolutional features output by all convolutional branches in the Mth stage (the last stage) can be obtained, and multiple initial convolutional features can be fused. Then, the fused convolutional features are convolved with the same resolution as each convolutional branch to obtain the second target convolutional features with the corresponding resolution.

[0154] Specifically, in the process of fusing multiple initial convolutional features, each initial convolutional feature can be upsampled or downsampled according to the resolution relationship between each convolutional branch and the other convolutional branches, and added to the feature set to be fused of the corresponding resolution convolutional branch in the Mth stage. Then, all the initial convolutional features in the feature set to be fused are convolved at the corresponding resolution to generate a second target convolutional feature with the same resolution as each convolutional branch.

[0155] In one possible implementation, during the process of fusing the first image features and the second image features to obtain the target image features, the feature values ​​of each pixel in the first image features can be concatenated with the feature values ​​of the corresponding pixels in the second image features to obtain the third image features; then, the third image features are dimensionality reduced to obtain the fourth image features; and finally, the fourth image features are activated to obtain the target image features.

[0156] Referring to Figure 11, which is a schematic diagram of the target image feature generation process provided in an embodiment of this application, the difference image is a grayscale image, i.e., a single-channel image, while the image to be tested is a color RGB image, i.e., a three-channel image. In order to reuse the feature extraction module and accelerate training convergence, the single-channel data of the difference image can be copied first to form a three-channel image, so that the number of channels of the difference image is equal to the number of channels of the image to be tested. Then, the difference image of the three-channel data and the image to be tested are respectively input into the same feature extraction module in the defect detection model to obtain the first image feature and the second image feature, respectively. The dimension of the first image feature is the same as the dimension of the second image feature, both being 256.

[0157] After concatenating the first image feature with the second image feature, a third image feature is obtained. The concatenation operation is equivalent to splicing the two feature tensors along the channel dimension (or feature dimension), which can fuse and retain the information contained in the first image feature and the second image feature.

[0158] Since the dimension of the third image feature obtained by fusion is 256*2, and the third image feature contains redundant information, a 1*1 convolution kernel can be used to reduce the dimensionality of the third image feature through the channels to obtain a fourth image feature with a dimension of 256. This reduces unnecessary feature dimensions, compresses feature representation, and reduces the computational cost of subsequent processing. At the same time, mapping high-dimensional features to a lower-dimensional space can effectively extract higher-level feature representations from the third image feature, which is conducive to extracting more global and abstract information and improving the quality and reliability of the fourth image feature.

[0159] Then, the fourth image feature is activated using an activation function to obtain the target image feature. That is, the number of channels of the fourth image feature and the target image feature is the same as the number of channels of the image to be tested.

[0160] In one possible implementation, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. In the process of detecting defects in the image under test based on the target image features and obtaining the defect detection box information in the image under test, the target image features can first be input into the region proposal network for region extraction to obtain the reference detection box coordinates.

[0161] For the first network head, the reference detection box coordinates and the target image features are input to the local region feature extractor connected to the first network head for pooling. The resulting pooled features are input to the first network head for defect detection, and the defect detection box coordinates are output.

[0162] For each of the remaining network heads, the defect detection box coordinates output by the previous network head and the target image features are input to the local region feature extractor connected to that network head for pooling. The resulting pooled features are then input to that network head for defect detection, and the defect detection box coordinates are output.

[0163] Specifically, the coordinates of the defect detection box output by the last network header are determined as the defect detection box information.

[0164] Specifically, pooling can employ Region of Interest Pooling (ROIPooling) technology, which can extract feature regions of a specific size, extract convolutional feature maps corresponding to different sizes of Regions of Interest (ROIs) from the target image features based on feature mapping relationships, and pool them to form feature maps of the same dimension.

[0165] A Region Proposal Network (RPN) is a fully convolutional network that takes an image of arbitrary size as input and outputs a set of rectangular target proposals, each with a target score. Therefore, the RPN can be used to predict the boundaries of defect detection boxes and their confidence levels based on target image features.

[0166] Specifically, the target image features are input into the region proposal network to extract the possible defect regions in the target image features and output the coordinates of the reference detection box.

[0167] After obtaining preliminary reference detection box coordinates using the region proposal network, the network head in subsequent stages can use these preliminary reference detection box coordinates to optimize and obtain more detailed and accurate defect detection box coordinates.

[0168] All network heads other than the first one can receive the output of the previous network head. The multi-cascade method allows the network heads to continuously learn and extract higher-level feature representations, further optimize and refine the defect detection box, and improve the accuracy of the defect detection box.

[0169] Referring to Figure 12, which is a schematic diagram of the defect detection process provided in an embodiment of this application, the defect detection model includes multiple cascaded network heads. As shown in Figure 12, the defect detection model may include network head H1, network head H2, and network head H3, as well as a local region feature extractor. Network head H1 is the first network head, and network head H3 is the last network head.

[0170] First, the target image features (i.e., image features P) are input into the Region Proposal Network (RPN) for region extraction to obtain the reference frame coordinates B0.

[0171] Next, the reference detection box coordinates B0 and image features P are input to the first local region feature extractor for region-of-interest pooling processing to extract pooling features, which are then fed into the first network head H1. Network head H1 can adjust the position and size of the defect detection box based on the reference detection box coordinates B0, and then output the defect detection box coordinates B1. Network head H1 can also classify defect types and output the classification result C1 for the defect detection box coordinates B1. The classification result C1 indicates the defect type corresponding to the defect detection box coordinates B1.

[0172] Then, the defect detection box coordinates B1 and image features P are input to the second local region feature extractor for region of interest pooling processing. The pooling features of this stage are extracted and sent to the next network head H2 for defect detection. The position and size of the defect detection box are further adjusted, and a new defect detection box coordinate B2 is output. At the same time, the network head H2 can also output the classification result C2 to indicate the defect type of the defect detection box coordinates B2.

[0173] Next, the defect detection box coordinates B3 and the image feature P are input into the third local region feature extractor for region-of-interest pooling. After extracting new pooled features, these are input into the final network head H3. Network head H3 performs a final defect detection based on the pooled features, readjusts the position and size of the defect detection box, and outputs a new defect detection box coordinate B3. This coordinate B3 is then used as the defect detection box information for the image under test. Additionally, network head H3 can also detect the classification result C3 corresponding to the defect detection box coordinates B3, thus the defect detection box information for the image under test can also include the classification result of the defect type corresponding to the defect detection box.

[0174] Since the detection results at each stage will affect the defect detection effect in the next stage, the iterative approach allows the model to gradually improve the detection results. That is, the reference defect detection box is first roughly located through the Region Proposal Network (RPN), and then the position and size of the defect detection box are continuously refined through multiple cascaded network heads to improve the detection accuracy.

[0175] In one possible implementation, the defect detection box information includes the defect detection box coordinates, the detection box confidence probability, and the target defect category. During the process of determining the second detection result of the target workpiece based on the defect detection box information and the first detection result, when the first detection result indicates that the target workpiece has no defects, a preset confidence probability threshold and a preset area threshold corresponding to the target defect category are obtained. Next, the defect area is determined based on the defect detection box coordinates, and a first relationship between the defect area and the preset area threshold, as well as a second relationship between the detection box confidence probability and the preset confidence probability threshold, are determined. Finally, the second detection result of the target workpiece is determined based on the first and second relationships.

[0176] When the first detection result indicates that there is no defect in the target workpiece, it means that based on the differential image, it is determined that there is no visually obvious defect in the target workpiece. Therefore, the defect detection box information output by the defect detection model can be filtered by using the preset confidence probability threshold and preset area threshold to further determine whether there is a defect in the target workpiece, thereby reducing the occurrence of false alarms and false negatives.

[0177] In addition to outputting the coordinates of the defect detection boxes, the defect detection model also outputs the target defect category (i.e., the classification result in Figure 12) and the confidence probability of the detection boxes corresponding to each defect detection box coordinate.

[0178] The target workpiece can have multiple defect categories, such as dirt, discoloration, scratches, damage, and dents. The quality inspection standards differ for different defect categories. Therefore, it is necessary to obtain a preset confidence probability threshold and a preset area threshold (i.e., the quality inspection standard) based on the target defect category corresponding to each defect detection frame. By comparing the first relationship between the defect area of ​​the defect detection frame and the preset area threshold, and the second relationship between the detection confidence probability and the preset confidence probability threshold, it can be determined whether the defect detection frame contains defects that cannot pass quality inspection.

[0179] The first size relationship is used to indicate whether the defect area exceeds a preset area threshold, which can reflect the quality of the target workpiece. If the defect area is greater than or equal to the preset area threshold, the quality of the target workpiece can be considered substandard, and the target workpiece has a defect.

[0180] The second magnitude relation is used to indicate the probability of a defect detection box, reflecting the likelihood that the target workpiece contains the defect. If the confidence probability of the detection box is greater than a preset confidence probability threshold, it can be considered that the target workpiece contains a defect at the location of the corresponding defect detection box.

[0181] The second inspection result of the target workpiece is determined based on the first and second size relationships. The judgment logic can be adjusted according to the actual quality inspection situation. For example, if the first size relationship is that the defect area is greater than or equal to the preset area threshold, or the second size relationship is that the confidence probability of the detection frame is greater than the preset confidence probability threshold, then the second inspection result of the target workpiece can be determined to be that the target workpiece has a defect and cannot pass the quality inspection. Or, if the first size relationship is that the defect area is greater than or equal to the preset area threshold, and the second size relationship is that the confidence probability of the detection frame is greater than the preset confidence probability threshold, then the second inspection result of the target workpiece can be determined to be that the target workpiece has a defect.

[0182] Specifically, as shown in Figure 13, which is a schematic diagram of the second detection result determination process provided in the embodiment of this application, when the first detection result indicates that the target workpiece has no defects, the defect detection box information output by the defect detection model is obtained, specifically including the defect detection box coordinates K1, K2, and K3. The confidence probability P1 of the detection box corresponding to defect detection box coordinate K1 is 0.8, and the corresponding target defect category is dirt; the confidence probability P2 of the detection box corresponding to defect detection box coordinate K2 is 0.3, and the corresponding target defect category is scratches; the confidence probability P3 of the detection box corresponding to defect detection box coordinate K3 is 0.5, and the corresponding target defect category is dents.

[0183] Next, preset confidence probability thresholds and preset area thresholds are obtained for the defect categories of dirt, scratches, and dents, respectively. Specifically, the preset dirt confidence probability threshold for dirt is 0.7, and the preset dirt area threshold is 15cm². 2 The preset confidence probability threshold for scratch damage is 0.4, and the preset scratch damage area is 3cm². 2 The preset confidence probability threshold for the indentation is 0.7, and the preset indentation area is 10 cm². 2 .

[0184] Then, based on the coordinates of each defect detection frame, the corresponding defect area is determined. Specifically, the area S1 of the dirt defect calculated from the defect detection frame coordinates K1 is 20 cm². 2 The scratch defect area S2 calculated from the defect detection frame K2 is 3.5 cm². 2 The area S3 of the dent defect calculated from the coordinates K3 of the defect detection frame is 2 cm. 2 .

[0185] By comparing the defect area of ​​each defect category with the preset area threshold, and the detection frame confidence probability with the preset confidence probability threshold, it can be found that for the dirt category, the first size relationship is that the dirt defect area is greater than the preset dirt area threshold, and the second size relationship is that the dirt detection frame confidence probability is greater than the preset dirt confidence probability. Therefore, the defect marked at defect detection frame coordinate K1 is considered a defect that cannot pass quality inspection. For the dent category, the first size relationship is that the dent defect area is less than the preset dent area threshold, and the second size relationship is that the scratch damage detection frame confidence probability is less than the preset scratch damage confidence probability. Therefore, the defects marked at defect detection frame coordinates K2 and K3 can be considered defects that can pass quality inspection. However, since the target workpiece contains a defect that cannot pass quality inspection, namely the defect marked at defect detection frame coordinate K1, it can be determined that the second inspection result of the target workpiece is that the target workpiece has a defect and cannot pass quality inspection.

[0186] In one possible implementation, when the first detection result indicates that the target workpiece has a defect, the second detection result of the target workpiece is determined to be that the target workpiece has a defect. If the first detection result indicates that the target workpiece has a defect, it means that the target workpiece already has a visually obvious defect, and no further judgment is needed. The second detection result of the target workpiece can be directly determined to be that the target workpiece has a defect, thereby improving the efficiency of defect detection.

[0187] Defect detection of the target workpiece can be performed using both comparative and model prediction methods.

[0188] The target workpiece first obtains a first detection result based on a comparison method. If the first detection result indicates that the target workpiece has a defect, the second detection result indicating that the target workpiece has a defect can be directly obtained without considering the defect detection box information obtained by the model prediction method.

[0189] If the reference defect result indicates that the target workpiece does not have a defect, it is necessary to obtain the defect detection box information output by the defect detection model. Then, based on the defect detection box coordinates, detection box confidence probability, and target defect category in the defect detection box information, it is compared with the preset confidence probability threshold and preset area threshold corresponding to the target defect category to determine the target reference defect result.

[0190] In one possible implementation, during the process of comparing the image to be tested with the template image to obtain a difference image, the transformation parameters between the image to be tested and the template image can be determined first; according to the transformation parameters, the target workpiece displayed in the template image is aligned with the target workpiece displayed in the image to be tested to obtain an aligned image to be tested; the aligned image to be tested is compared with the template image to obtain an initial difference image; the initial difference image is filtered to obtain the difference image.

[0191] Since the images of the target workpieces to be tested are usually obtained by taking pictures with camera equipment, the shooting position or placement direction of each target workpiece is prone to deviation. Moreover, the images to be tested and the template images are obtained under different conditions such as different times and different angles. As a result, it is difficult to align the target workpieces shown in the images to be tested with the target workpieces shown in the template images, so as to avoid affecting the comparison effect.

[0192] Therefore, by aligning the image to be tested with the template image, the target workpiece displayed in the image to be tested can be aligned with the target workpiece displayed in the template image, thereby improving the comparison effect and the accuracy of the initial difference image.

[0193] Specifically, the transformation parameters between the image to be tested and the template image are first determined, and the image to be tested and the template image are aligned to the same coordinate space, so that pixels at the same coordinate position correspond to the same actual position, thereby ensuring that the content of the same area is compared during the comparison process.

[0194] Determining the transformation parameters and aligning the images can achieve spatial consistency between the image under test and the template image, which helps to effectively extract the information from the two images and eliminate noise caused by framing differences, thereby helping to obtain a more accurate initial difference image.

[0195] Referring to Figure 14, which is a schematic diagram of the effect of morphological filtering provided in the embodiment of this application. Even if the image to be tested and the template image are aligned, the edge regions of both inevitably have misaligned parts, which are prone to noise and interfere with the generation of the difference image. Therefore, edge noise can be removed by performing morphological image processing filtering (such as dilation, erosion, filtering, smoothing, etc.) on the initial difference image.

[0196] As shown in sub-image (a) of Figure 14, an erosion operation can be performed on the initial difference image to erode or shrink the edges in the initial difference image, resulting in the eroded effect shown in sub-image (b) of Figure 14. This involves replacing each pixel f(x,y) in the initial difference image f with the minimum value in set B (adjacent pixels). The erosion operation on the initial difference image f is performed using set B, and the specific formula is as follows:

[0197] In another embodiment, the edges in the initial difference image can be dilated (or expanded) to obtain the dilated effect shown in sub-image (c) of Figure 16. That is, each pixel f(x,y) in the initial difference image f is replaced with the maximum value in set C (neighboring pixels), and the dilation operation of the initial difference image f is performed using set C. The specific formula is as follows:

[0198] After morphological filtering, noise at the edges of the initial difference image can be removed to obtain the difference image. The difference image and the image to be tested are input into the defect detection model, which can assist the model in feature extraction and defect detection. Thus, the model can accurately locate the defect region image shown in sub-image (d) in Figure 14. The defect region image is part of the image information in the defect detection box information output by the model.

[0199] In one possible implementation, during the process of determining the transformation parameters between the image to be tested and the template image, a sliding window can be used to determine a current window region in the image to be tested; calculate the correlation coefficient between the image within the current window region and the image within the corresponding window region in the template image; determine a target region in the image to be tested based on the correlation coefficients of each window region; determine the transformation matrix required to transform the target region to the aligned region in the template image; and determine the inverse of the transformation matrix as the transformation parameters.

[0200] Referring to Figure 15, which is a schematic diagram of the process for determining the correlation coefficient provided in an embodiment of this application, the sliding window method refers to sliding the image to be tested according to the size of the window and calculating the correlation coefficient between the corresponding window regions in the template image and the image to be tested each time. This can measure the similarity between the two, thereby finding the region in the image to be tested that is most similar to the template image, i.e., the window region with the highest correlation coefficient, as the target region.

[0201] Once the target region is known, the transformation matrix can be determined. The transformation matrix describes the geometric transformations (such as translation, rotation, or distortion) required to convert the target region in the test image into a region aligned with the template image. The transformation matrix M can be as follows:

[0202] Wherein, in the transformation matrix M, (t) x , t y Let ) represent the displacement vector. A rigid body transformation is achieved by adding a rotation angle θ to the displacement vector. A similarity transformation is achieved by adding a scaling transformation parameter s to the displacement and rotation. The similarity transformation changes the distance between points before and after the transformation, but keeps the angle unchanged.

[0203] As shown in Figure 16, Figure 16 is a schematic diagram of the similarity transformation correspondence between the template image and the image to be tested provided in the embodiments of this application. After determining the transformation matrix M, it is also necessary to calculate the inverse matrix M of the transformation matrix M. inv Inverse matrix M inv It can be applied to convert template images into target regions in the image to be tested, that is, transform the position of the target workpiece displayed in the template image to the position of the target workpiece displayed in the target region of the image to be tested, thereby achieving workpiece alignment.

[0204] The correlation coefficient between corresponding regions in the template image and the test image can be calculated based on the relative values ​​of pixel values ​​and their mean values ​​in the template image, and the relative values ​​of pixel values ​​and their mean values ​​in the corresponding regions of the test image. A higher correlation coefficient indicates a better match between the corresponding regions in the test image and the template image. The formula for calculating the correlation coefficient is shown below:

[0205] Where T′(x′,y′) represents the relative value of the pixel value at coordinate (x′,y′) in the template image to the mean pixel value of the template image, and I′(x+x′,y+y′) represents the relative value of the pixel value at coordinate (x+x′,y+y′) in the corresponding window region of the image to be tested to the mean pixel value of the window region.

[0206] In one possible implementation, the template image and the image to be tested can be converted to grayscale separately. Then, based on the grayscale values ​​of the sliding window region in the image to be tested and the template image, the difference, mean absolute difference, sum of absolute errors, sum of squared average errors, and normalized product are calculated as correlation coefficients to determine the target region.

[0207] In one possible implementation, a deep learning registration algorithm can be used to perform image matching between the sliding window region in the image to be tested and the template image, such as the feature matching algorithm based on graph convolutional neural networks (SuperGlue) and the self-supervised feature point detection algorithm based on deep learning (SuperPoint).

[0208] In one possible implementation, referring to Figure 17, which is a schematic diagram of the training process of the defect detection model provided in the embodiment of this application, the defect detection model can be trained through the following steps 1701 to 1705:

[0209] Step 1701: Obtain the training image of the target workpiece, compare the training image with the template image to obtain the sample difference image, wherein the training image is labeled with the corresponding defect category label;

[0210] Step 1702: Input the training image and the sample difference image into the defect detection model to obtain the sample detection box information in the training image, wherein the sample detection box information includes the sample defect category of the target workpiece;

[0211] Step 1703: Determine the initial classification loss based on the sample defect category and defect category label;

[0212] Step 1704: Adjust the initial classification loss according to the preset first adjustment parameter and second adjustment parameter to obtain the target classification loss. The first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the classification difficulty corresponding to the defect category label.

[0213] Step 1705: Train the defect detection model based on the target classification loss.

[0214] Specifically, by comparing training images with template images, the defect features of the target workpiece can be learned better. Sample difference images are helpful in highlighting the difference between defect areas and normal areas. Then, the training images and sample difference images are input into the defect detection model. The sample difference images provide differential information to the model, which helps the model learn and train, and improves the model's generalization ability.

[0215] The initial classification loss can be determined based on the cross-entropy loss function, using the sample defect category and defect category label.

[0216] Next, by presetting the first adjustment parameter and the second adjustment parameter, the initial classification loss can be adjusted using the first adjustment parameter and the second adjustment parameter to obtain the target loss function. Specifically, the target loss function can be represented by the following formula:

[0217] Where x represents the predicted score vector output by the defect detection model, class represents the defect category label, x[class] represents the original predicted score of the defect detection model for the current output defect belonging to a specific category, and α class Let γ be the first adjustment parameter, and let γ be the second adjustment parameter. This represents the cross-entropy operation process, which converts the model's original output into a probabilistic form to obtain the predicted probability that the current output defect belongs to a specific category. The higher the predicted probability, the closer the current output defect is to the specific category, and the more accurate the classification.

[0218] Compared to the cross-entropy loss function, the objective loss function adds a modulation factor. For accurately classified samples, the predicted probability approaches 1, and the modulation factor approaches 0; for inaccurately classified samples, the modulation factor approaches 0. Therefore, for the cross-entropy loss function, the target loss function remains unchanged for inaccurately classified samples, but decreases for accurately classified samples. Thus, overall, it's equivalent to increasing the weight of inaccurately classified samples in the target loss function. Simultaneously, the output predicted probability reflects the difficulty of the model's classification; a higher predicted probability indicates higher confidence in the model's classification, meaning the samples are easier to classify; conversely, a lower predicted probability indicates lower confidence in the model's classification, meaning the samples are more difficult to classify.

[0219] Therefore, the objective loss function can increase the weight of difficult-to-distinguish samples in the objective loss function, making the objective loss function bias towards difficult-to-distinguish samples, which helps to improve the accuracy of classifying difficult samples.

[0220] Since training images of defect-free target workpieces are easy to obtain, while defective target workpieces are difficult to obtain, and the number of defect samples required increases with the number of defect types, the first adjustment parameter can suppress the imbalance between the number of positive and negative samples, and the second adjustment parameter can control the imbalance between the number of simple / difficult-to-distinguish samples. Positive samples can be represented as sample images of target workpieces with defects, and negative samples can be represented as sample images of target workpieces without defects.

[0221] According to the above formula (5), the first adjustment parameter can be used to adjust the loss ratio between positive and negative samples, while the second adjustment parameter can be used to adjust the loss ratio between easy and difficult samples. This can solve the class imbalance problem in object detection, reduce the weight of easy-classified samples, and make the model focus more on difficult samples during training.

[0222] In one possible implementation, the second adjustment parameter may include a first decoupling factor and a second decoupling factor, wherein the first decoupling factor is a base value used to adjust the loss ratio between samples of different classification difficulty, and the second decoupling factor is a variable parameter related to the imbalance of samples in each class.

[0223] Specifically, the second adjustment parameter γ in formula (5) can be referred to. Where, γ b Represented as the first decoupling factor, Let g be the second decoupling factor, j be the j-th class sample, and g be the second decoupling factor. j G represents the training balance coefficient of the j-th class sample. j The larger the value of g, the more balanced the training of the j-th class of samples. jThe smaller the value, the more imbalanced the training is for that category. s represents the scaling factor that determines the upper limit of the second adjustment parameter in the target loss function, which is used to help adjust the imbalance of the number of simple / difficult-to-distinguish samples.

[0224] Therefore, by decoupling the second adjustment parameter, a first decoupling factor and a second decoupling factor are formed. These two factors can then be used independently to address the imbalance problem in each class of samples, thereby improving model performance. Specifically, taking a total number of sample classes as C as an example, the objective loss function combining the first adjustment parameter, the first decoupling factor, and the second decoupling factor can be expressed as follows:

[0225] In one possible implementation, during the training of the defect detection model, the sample detection box information includes the defect detection box and the corresponding detection box confidence. After obtaining the sample box information, a non-maximum suppression process can be performed based on the detection box confidence. The detection boxes are sorted according to their respective confidence. The difficult samples (defect detection boxes with low confidence) are input into the next stage of training for forward and backward propagation. This allows the model to obtain more targeted information during supervised training, improving the speed and quality of training.

[0226] Specifically, the Intersection over Union (IoU) ratio of each defect detection box is calculated, which is the ratio of the intersection to the union of each defect detection box with the defect detection box that has the highest confidence level. Based on the IoU ratio of each defect detection box, the confidence level of the corresponding detection box is adjusted. If the IoU ratio exceeds a preset threshold, the confidence level of the corresponding detection box is reduced; if the IoU ratio does not exceed the preset threshold, the confidence level of the corresponding detection box is not adjusted. Specifically, the formula for adjusting the detection box confidence level is as follows:

[0227] Where, N t Represented as a preset ratio threshold, IoU(M,b) i ) represents the intersection-union ratio of the defect detection boxes, M represents the defect detection box with the highest confidence, and b i Represented as the current defect detection bounding box, s i This is represented by the current defect detection box b. i The detection frame confidence is high enough to retain overlapping, inconspicuous defects, thus reducing the probability of missed or false detections.

[0228] In one possible implementation, during the training of the defect detection model based on the target classification loss, the coordinate difference of the defect detection box can be determined based on the coordinates of the defect detection box in the sample detection box information and the coordinates of the actual label detection box marked in the training image; then, the regression loss of the defect detection box is determined based on the coordinate difference; and then, the defect detection model is trained based on the regression loss and the target classification loss.

[0229] Therefore, by combining regression loss and target classification loss to train the defect detection model, the regression loss is used to fit the coordinates of the defect detection box, and the target classification loss is used to supervise and constrain the category of the defect detection box. This simultaneously guides the defect detection model in both defect detection box regression and classification, improves the performance of the defect detection model, and alleviates the impact of class imbalance in the defect detection model.

[0230] In one possible implementation, the sample detection box information in the training image also includes the coordinates of the defect detection box. Therefore, the difference between the defect detection box predicted by the model and the actual label detection box in the training image in coordinates (x, y, w, h) can be calculated. Here, the x element represents the horizontal coordinate of the center of the detection box, the y element represents the vertical coordinate of the center of the detection box, the w element represents the width of the detection box, i.e., the size along the horizontal direction of the image, and the h element represents the height of the detection box, i.e., the size along the vertical direction of the image.

[0231] Specifically, the defect detection bounding box and the actual label detection bounding box can be normalized separately to eliminate the influence of size and obtain a more accurate coordinate difference. Next, a regression loss function, such as the Smooth L1 Loss function, can be used to calculate the regression loss of the defect detection bounding box based on the coordinate difference. Then, the regression loss can be used to fit and regress the detection bounding box.

[0232] Specifically, the regression loss can be optimized using gradient descent. During the optimization process, the model parameters are continuously updated so that the defect detection boxes predicted by the model gradually approach the actual label detection boxes. By minimizing the regression loss, the defect detection model can fit more accurate defect detection boxes.

[0233] In one possible implementation, referring to Figure 18, which is a schematic diagram of a defect detection process provided in another embodiment of this application, the defect detection process for the target workpiece may include two branch processes, one of which may include the following steps:

[0234] 1801. Obtain the template image and the image to be tested of the target workpiece.

[0235] 1802. Align the image to be tested with the template image and generate a transformation matrix.

[0236] 1803. Rotate and translate the template image to align it with the image to be tested.

[0237] 1804. Compare the image to be tested with the template image to generate a difference image.

[0238] 1805. Perform morphological image processing on the difference image, such as dilation / erosion, and filter out noise regions (differential outliers caused by slight workpiece offset).

[0239] 1806. Based on a logical algorithm, locate the defect region in the differential image and determine the first detection result.

[0240] Another branch of the process may include the following steps:

[0241] 1807. The image features of the difference image are concatenated with the image features of the image to be tested to form four-channel image feature data.

[0242] 1808. Use the defect detection model to perform defect detection and output the defect detection box information.

[0243] 1809. Combine the first inspection result and the defect detection box information for joint post-processing to determine the second inspection result of the target workpiece. Specifically, in the step of determining the second inspection result of the target workpiece by combining the first inspection result and the defect detection box information, the first inspection result and the defect detection box information can be input into the joint post-processing model for quality inspection prediction, and the second inspection result of the target workpiece can be output.

[0244] In one possible implementation, during the process of inputting the image to be tested and the difference image into the defect detection model to extract the target image features, the image to be tested and the difference image can be concatenated to obtain a concatenated image. Then, the concatenated image is input into the defect detection model to extract the target image features from the concatenated image.

[0245] In one possible implementation, the image to be tested and the difference image are first scaled to ensure that the corresponding dimensions of the two images are consistent in the concatenation direction. The concatenation of the image to be tested and the difference image can be horizontal or vertical, that is, the two images are stitched together along the horizontal direction or along the vertical direction.

[0246] Referring to Figure 19, which is a schematic diagram of the defect detection frame information generation process provided in another embodiment of this application, the image to be tested is a three-channel color RGB image, while the differential image can be a single-channel grayscale image. The image to be tested and the differential image are directly concatenated to form a concatenated image, wherein the concatenated image is a four-channel image.

[0247] Since there is no need to extract features from the test image and the difference image separately, the loss caused by feature extraction is reduced. The concatenated images can retain the original feature information of the two images, which helps the model to learn and extract more accurate target image features. At the same time, the input of the concatenated images into the defect detection model can provide the difference information of different images at the same position, which helps to improve the accuracy of the model detection.

[0248] Since the target image features are extracted based on concatenated images, which are generated by stitching together the image to be tested and the difference image, and the difference image carries the difference information between the target workpiece and the reference workpiece, the target image features can capture key difference information. Therefore, the target image features can be used to assist the model in defect detection and improve the accuracy of the defect detection box information.

[0249] The defect detection method provided in the embodiments of this application is described in detail below.

[0250] Referring to Figure 20, which is a schematic diagram of an optional overall process of a defect detection method provided in an embodiment of this application, the defect detection method includes, but is not limited to, the following steps 2001 to 2024:

[0251] Step 2001: Obtain the image to be tested of the target workpiece and the template image of the target workpiece.

[0252] Step 2002: Use a sliding window to determine the current window region in the image to be tested; calculate the correlation coefficient between the image in the current window region and the image in the corresponding window region of the template image.

[0253] Step 2003: Determine the target region in the image to be tested based on the correlation coefficient of each window region; determine the transformation matrix required to transform the target region to the region aligned with the template image.

[0254] Step 2004: Determine the inverse matrix of the transformation matrix and use the inverse matrix as the transformation parameter between the image to be tested and the template image.

[0255] Step 2005: According to the transformation parameters, align the target workpiece displayed in the template image with the target workpiece displayed in the image to be tested to obtain the aligned image to be tested.

[0256] Step 2006: Compare the aligned image to be tested with the template image to obtain the initial difference image.

[0257] Step 2007: Perform morphological filtering on the initial difference image to obtain the difference image.

[0258] Step 2008: Copy the data of each channel in the difference image until the number of channels in the copied difference image is equal to the number of channels in the image under test.

[0259] Step 2009: Input the image to be tested and the copied difference image into the defect detection model respectively.

[0260] In this step, the image to be tested and the copied differential image share the same feature extraction module in the defect detection model.

[0261] Step 2010: Perform multiple stages of convolution on the image to be tested sequentially.

[0262] In this step, each stage includes at least one convolutional branch, and each convolutional branch is used to perform multiple convolutions. When moving from the current stage to the next stage, a new convolutional branch is added. The resolution of the new convolutional branch is smaller than that of any existing convolutional branch in the current stage. The feature input to any convolutional branch in the current stage is obtained by fusing the features output by all convolutional branches in the previous stage.

[0263] Step 2011: Obtain the initial convolutional features output by each convolutional branch in the last stage, fuse multiple initial convolutional features to obtain fused features; for each convolutional branch in the last stage, perform convolution with the same resolution as the convolutional branch on the fused features to obtain the second target convolutional feature of the convolutional branch.

[0264] In this step, during the last convolution, for any second convolution branch, the features of the current convolution output are upsampled or downsampled according to the resolution relationship with the other second convolution branches, and then added to the input of the last convolution of the other second convolution branches; then the initial convolution features of the last convolution output of each second convolution branch are obtained.

[0265] Step 2012: Fuse multiple second target convolutional features to obtain the first target convolutional feature.

[0266] Step 2013: Downsample the first target convolutional features to obtain the first image features of the image to be tested.

[0267] Step 2014: Concatenate the feature values ​​of each pixel in the first image feature with the feature values ​​of the corresponding pixels in the second image feature to obtain the third image feature.

[0268] Step 2015: Dimensionally reduce the third image features to obtain the fourth image features.

[0269] Step 2016: Activate the fourth image feature to obtain the target image feature.

[0270] Step 2017: Perform defect detection on the image under test based on the features of the target image to obtain the defect detection box information in the image under test.

[0271] In this step, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The target image features are input to the region proposal network for region extraction to obtain reference detection box coordinates. For the first network head, the reference detection box coordinates and the target image features are input to the local region feature extractor connected to the first network head for pooling. The resulting pooled features are then input to the first network head for defect detection, and the defect detection box coordinates are output. For each of the remaining network heads, the defect detection box coordinates output by the previous network head and the target image features are input to the local region feature extractor connected to that network head for pooling. The resulting pooled features are then input to that network head for defect detection, and the defect detection box coordinates are output. The defect detection box coordinates output by the last network head are determined as the defect detection box information.

[0272] Step 2018: Locate the defect area in the differential image and determine the first detection result of the target workpiece based on the defect area.

[0273] Step 2019: Determine whether the first detection result indicates that the target workpiece has a defect. If yes, proceed to step 2020; otherwise, proceed to step 2021.

[0274] Step 2020: Determine that the second inspection result of the target workpiece is that the target workpiece has a defect, and proceed to step 2024.

[0275] Step 2021: Obtain the preset probability threshold and preset area threshold corresponding to the target defect category.

[0276] Step 2022: Determine the defect area based on the defect detection box coordinates, determine the first relationship between the defect area and the preset area threshold, and the second relationship between the detection box confidence probability and the preset confidence probability threshold.

[0277] Step 2023: Determine the second detection result of the target workpiece based on the first size relationship and the second size relationship.

[0278] Step 2024: End of process.

[0279] The defect detection method provided by another embodiment of this application is described in detail below.

[0280] Referring to Figure 21, which is a schematic diagram of an optional overall process of a defect detection method provided in an embodiment of this application, the defect detection method includes, but is not limited to, the following steps 2101 to 2120:

[0281] Step 2101: Obtain the image to be tested of the target workpiece and the template image of the target workpiece.

[0282] Step 2102: Use a sliding window to determine the current window region in the image to be tested; calculate the correlation coefficient between the image in the current window region and the image in the corresponding window region of the template image.

[0283] Step 2103: Determine the target region in the image to be tested based on the correlation coefficient of each window region; determine the transformation matrix required to transform the target region to the region aligned with the template image.

[0284] Step 2104: Determine the inverse matrix of the transformation matrix and use the inverse matrix as the transformation parameter between the image to be tested and the template image.

[0285] Step 2105: According to the transformation parameters, align the target workpiece displayed in the template image with the target workpiece displayed in the image to be tested to obtain the aligned image to be tested.

[0286] Step 2106: Compare the aligned image to be tested with the template image to obtain the initial difference image.

[0287] Step 2107: Perform morphological filtering on the initial difference image to obtain the difference image.

[0288] Step 2108: Concatenate the image to be tested with the difference image to obtain the concatenated image.

[0289] Step 2109: Input the concatenated image into the defect detection model and perform multi-stage convolution on the concatenated image.

[0290] In this step, each stage includes at least one convolutional branch, and each convolutional branch is used to perform multiple convolutions. When moving from the current stage to the next stage, a new convolutional branch is added. The resolution of the new convolutional branch is smaller than that of any existing convolutional branch in the current stage. The feature input to any convolutional branch in the current stage is obtained by fusing the features output by all convolutional branches in the previous stage.

[0291] Step 2110: Obtain the initial convolutional features output by each convolutional branch in the last stage, fuse multiple initial convolutional features to obtain fused features; for each convolutional branch in the last stage, perform convolution with the same resolution as the convolutional branch on the fused features to obtain the second target convolutional feature of the convolutional branch.

[0292] In this step, during the last convolution, for any second convolution branch, the features of the current convolution output are upsampled or downsampled according to the resolution relationship with the other second convolution branches, and then added to the input of the last convolution of the other second convolution branches; then the initial convolution features of the last convolution output of each second convolution branch are obtained.

[0293] Step 2111: Fuse multiple second target convolutional features to obtain the first target convolutional feature.

[0294] Step 2112: Downsample the first target convolutional features to obtain the target image features of the concatenated image.

[0295] Step 2113: Perform defect detection on the concatenated image based on the features of the target image to obtain the defect detection box information in the image to be tested.

[0296] In this step, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The target image features are input to the region proposal network for region extraction to obtain reference detection box coordinates. For the first network head, the reference detection box coordinates and the target image features are input to the local region feature extractor connected to the first network head for pooling. The resulting pooled features are then input to the first network head for defect detection, and the defect detection box coordinates are output. For each of the remaining network heads, the defect detection box coordinates output by the previous network head and the target image features are input to the local region feature extractor connected to that network head for pooling. The resulting pooled features are then input to that network head for defect detection, and the defect detection box coordinates are output. The defect detection box coordinates output by the last network head are determined as the defect detection box information.

[0297] Step 2114: Locate the defect area in the differential image and determine the first detection result of the target workpiece based on the defect area.

[0298] Step 2115: Determine whether the first detection result indicates that the target workpiece has a defect. If yes, proceed to step 2116; otherwise, proceed to step 2117.

[0299] Step 2116: If the second inspection result of the target workpiece is determined to be that the target workpiece has a defect, proceed to step 2120.

[0300] Step 2117: Obtain the preset probability threshold and preset area threshold corresponding to the target defect category.

[0301] Step 2118: Determine the defect area based on the defect detection box coordinates, determine the first relationship between the defect area and the preset area threshold, and the second relationship between the detection box confidence probability and the preset confidence probability threshold.

[0302] Step 2119: Determine the second detection result of the target workpiece based on the first size relationship and the second size relationship.

[0303] Step 2120: End of process.

[0304] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0305] Referring to Figure 22, which is an optional structural schematic diagram of the defect detection device 2200 provided in an embodiment of this application, the defect detection device 2200 includes:

[0306] The first processing module 2201 is used to acquire the image to be tested of the target workpiece and the template image of the target workpiece, and compare the image to be tested with the template image to obtain a difference image;

[0307] The second processing module 2202 is used to input the image to be tested and the difference image into the defect detection model and perform the following processing: extracting target image features based on the image to be tested and the difference image; performing defect detection on the image to be tested based on the target image features to obtain defect detection box information in the image to be tested;

[0308] The third processing module 2203 is used to locate the defect region in the differential image and determine the first detection result of the target workpiece based on the defect region;

[0309] The fourth processing module 2204 is used to determine the second detection result of the target workpiece based on the defect detection box information and the first detection result.

[0310] Furthermore, the second processing module 2202 is used to: extract a first image feature from the image to be tested; extract a second image feature from the difference image; and fuse the first image feature and the second image feature to obtain the target image feature.

[0311] In one possible implementation, the second processing module 2202 is used for:

[0312] The data of each channel in the differential image is copied until the number of channels in the copied differential image is equal to the number of channels in the image under test.

[0313] The image to be tested and the copied difference image are respectively input into the defect detection model, wherein the image to be tested and the copied difference image share the same feature extraction module in the defect detection model.

[0314] In one possible implementation, the second processing module 2202 is used for:

[0315] Multiple stages of convolution are sequentially performed on the image to be tested, wherein,

[0316] Each stage includes at least one convolutional branch, and each convolutional branch is used to perform multiple convolutions;

[0317] When moving from the current stage to the next stage, a new convolutional branch is added, and the resolution of the new convolutional branch is less than that of any existing convolutional branch in the current stage.

[0318] The feature input to any convolutional branch in the current stage is obtained by fusing the features output by all convolutional branches in the previous stage;

[0319] The initial convolutional features output from each convolutional branch in the last stage are fused together to obtain the first target convolutional feature;

[0320] The first target convolutional features are downsampled to obtain the first image features.

[0321] In one possible implementation, the second processing module 2202 is used for:

[0322] Multiple initial convolutional features are fused to obtain the fused features;

[0323] For each of the multiple convolutional branches in the last stage, the fused features are subjected to a convolution with the same resolution as that convolutional branch to obtain the second target convolutional feature of that convolutional branch.

[0324] Multiple second target convolutional features are fused together to obtain the first target convolutional feature.

[0325] In one possible implementation, the second processing module 2202 is used for:

[0326] The feature values ​​of each pixel in the first image feature are concatenated with the feature values ​​of the corresponding pixels in the second image feature to obtain the third image feature;

[0327] The third image feature is dimensionality reduced to obtain the fourth image feature;

[0328] The fourth image feature is activated to obtain the target image feature.

[0329] In one possible implementation, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The second processing module 2202 is used for:

[0330] The target image features are input into the region proposal network for region extraction to obtain the coordinates of the reference detection box.

[0331] For the first network header

[0332] The reference detection box coordinates and the target image features are input into the local region feature extractor connected to the first network head for pooling. The resulting pooled features are input into the first network head for defect detection, and the defect detection box coordinates are output.

[0333] For each of the remaining network headers

[0334] The defect detection box coordinates output by the previous network head and the target image features are input to the local region feature extractor connected to the network head for pooling. The resulting pooled features are then input to the network head for defect detection, and the defect detection box coordinates are output.

[0335] Specifically, the coordinates of the defect detection box output by the last network header are determined as the defect detection box information.

[0336] In one possible implementation, the defect detection box information includes the defect detection box coordinates, the detection box confidence probability, and the target defect category. The fourth processing module 2204 is used for:

[0337] When the first detection result indicates that the target workpiece has no defects, the preset probability threshold and preset area threshold corresponding to the target defect category are obtained;

[0338] The defect area is determined based on the coordinates of the defect detection box, and a first relationship between the defect area and the preset area threshold is determined, as well as a second relationship between the confidence probability of the detection box and the preset confidence probability threshold.

[0339] The second detection result is determined based on the first size relationship and the second size relationship.

[0340] In one possible implementation, the fourth processing module 2204 is further configured to:

[0341] When the first detection result indicates that the target workpiece has a defect, the second detection result is determined to indicate that the target workpiece has a defect.

[0342] In one possible implementation, the first processing module 220 is used for:

[0343] Determine the transformation parameters between the image to be tested and the template image;

[0344] According to the transformation parameters, the target workpiece displayed in the template image is aligned with the target workpiece displayed in the image to be tested to obtain the aligned image to be tested;

[0345] The aligned image to be tested is compared with the template image to obtain an initial difference image;

[0346] The initial difference image is filtered to obtain the difference image.

[0347] In one possible implementation, the first processing module 2201 is used for:

[0348] Use a sliding window to determine the current window region in the image to be tested;

[0349] Calculate the correlation coefficient between the image within the current window region and the image within the corresponding window region of the template image;

[0350] Based on the correlation coefficients of each window region, the target region is determined in the image to be tested;

[0351] Determine the transformation matrix required to transform the target region to the region aligned in the template image;

[0352] The inverse of the transformation matrix is ​​used to determine the transformation parameters.

[0353] Furthermore, the second processing module 2202 is used for:

[0354] The image to be tested is concatenated with the difference image to obtain a concatenated image;

[0355] The concatenated images are input into the defect detection model to extract the features of the target image.

[0356] In one possible implementation, the defect detection device further includes a first training module, which is used for:

[0357] Acquire training images of the target workpiece, wherein the training images are labeled with corresponding defect category tags;

[0358] The training image is compared with the template image to obtain a sample difference image;

[0359] The training image and the sample difference image are input into the defect detection model to obtain sample detection box information in the training image, wherein the sample detection box information includes the sample defect category of the target workpiece;

[0360] Determine the initial classification loss based on the sample defect category and the defect category label;

[0361] The initial classification loss is adjusted according to the preset first adjustment parameter and second adjustment parameter to obtain the target classification loss. The first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the classification difficulty corresponding to the defect category label.

[0362] The defect detection model is trained based on the target classification loss.

[0363] The aforementioned defect detection device 2200 and defect detection method are based on the same inventive concept. They compare the target workpiece's image to be tested with a template image to obtain a difference image. This difference image provides visual information about the differences in the target workpiece. The image to be tested and the difference image are then input into the defect detection model, introducing difference information and improving the model's feature extraction accuracy. Next, target image features are extracted, and defect detection is performed on the target image based on these features, obtaining defect detection box information in the target image. By utilizing the features corresponding to the difference image in the target image features to highlight the defect-related features in the target image, the model's ability to detect defects lacking training samples and thus not learned is improved, increasing the accuracy of the defect detection box information. The defect region located through the difference image determines the first detection result of the target workpiece. The second detection result of the target workpiece is determined by combining the first detection result and the defect detection box information. In other words, by combining features obtained from multiple comparisons with deep learning algorithms, the high visual visibility of the compared features can visually assist in defect detection, effectively improving the accuracy of defect detection and reducing dependence on training samples.

[0364] The electronic device provided in this application embodiment for performing the above-described defect detection method can be a terminal. Referring to Figure 23, which is a partial structural block diagram of the terminal provided in this application embodiment, the terminal includes: a camera assembly 2310, a memory 2320, an input unit 2330, a display unit 2340, a sensor 2350, an audio circuit 2360, a wireless fidelity (WiFi) module 2370, a processor 2380, and a power supply 2390, etc. Those skilled in the art will understand that the terminal structure shown in Figure 23 does not constitute a limitation on the terminal, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0365] In this embodiment, the processor 2380 included in the terminal can execute the defect detection method of the previous embodiment.

[0366] The electronic device provided in this application embodiment for executing the above-described defect detection method can also be a server. Referring to Figure 24, which is a partial structural block diagram of the server provided in this application embodiment, the server 2400 can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 2422 (e.g., one or more processors) and a memory 2432, and one or more storage media 2430 (e.g., one or more mass storage devices) for storing application programs 2442 or data 2444. The memory 2432 and storage media 2430 can be temporary or persistent storage. The program stored in the storage media 2430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server 2400. Furthermore, the central processing unit 2422 may be configured to communicate with the storage media 2430 and execute the series of instruction operations in the storage media 2430 on the server 2400.

[0367] Server 2400 may also include one or more power supplies 2424, one or more wired or wireless network interfaces 2450, one or more input / output interfaces 2458, and / or one or more operating systems 2441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0368] The processor in the server 2400 can be used to execute defect detection methods.

[0369] This application also provides a computer-readable storage medium for storing program code for executing the defect detection methods of the foregoing embodiments.

[0370] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the defect detection method described above.

[0371] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this application, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0372] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0373] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0374] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0375] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0376] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0377] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0378] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0379] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A defect detection method, executed by an electronic device, comprising: The test image of the target workpiece and the template image of the target workpiece are acquired, and the test image and the template image are compared to obtain a difference image; The image to be tested and the difference image are input into the defect detection model, and the following processing is performed: Based on the image to be tested and the difference image, the features of the target image are extracted; Based on the features of the target image, defect detection is performed on the image to be tested to obtain defect detection box information in the image to be tested; The defect region is located in the differential image, and the first detection result of the target workpiece is determined based on the defect region; and, Based on the defect detection box information and the first detection result, a second detection result for the target workpiece is determined.

2. The defect detection method according to claim 1, wherein, The step of extracting target image features based on the image to be tested and the difference image includes: Extract the first image feature from the image to be tested; Extract the second image features from the difference image; The target image features are obtained by fusing the first image features and the second image features.

3. The defect detection method according to claim 2, wherein, The step of inputting the image to be tested and the difference image into the defect detection model includes: The data of each channel in the differential image is copied until the number of channels in the copied differential image is equal to the number of channels in the image under test. The image to be tested and the copied difference image are respectively input into the defect detection model, wherein the image to be tested and the copied difference image share the same feature extraction module in the defect detection model.

4. The defect detection method according to claim 2, wherein, The step of extracting the first image feature from the image to be tested includes: Multiple stages of convolution are sequentially performed on the image to be tested, wherein, Each stage includes at least one convolutional branch, and each convolutional branch is used to perform multiple convolutions; When moving from the current stage to the next stage, a new convolutional branch is added, and the resolution of the new convolutional branch is less than that of any existing convolutional branch in the current stage. The feature input to any convolutional branch in the current stage is obtained by fusing the features output by all convolutional branches in the previous stage; The initial convolutional features output from each convolutional branch in the last stage are fused together to obtain the first target convolutional feature; The first target convolutional features are downsampled to obtain the first image features.

5. The defect detection method according to claim 4, wherein, The process of fusing the initial convolutional features output from each convolutional branch in the last stage to obtain the first target convolutional feature includes: Multiple initial convolutional features are fused to obtain the fused features; For each of the multiple convolutional branches in the last stage, the fused features are subjected to a convolution with the same resolution as that convolutional branch to obtain the second target convolutional feature of that convolutional branch. Multiple second target convolutional features are fused together to obtain the first target convolutional feature.

6. The defect detection method according to claim 2, wherein, The process of fusing the first image features and the second image features to obtain the target image features includes: The feature values ​​of each pixel in the first image feature are concatenated with the feature values ​​of the corresponding pixels in the second image feature to obtain the third image feature; The third image feature is dimensionality reduced to obtain the fourth image feature; The fourth image feature is activated to obtain the target image feature.

7. The defect detection method according to any one of claims 1 to 6, wherein, The defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The step of performing defect detection on the image under test based on the target image features to obtain defect detection box information in the image under test includes: The target image features are input into the region proposal network for region extraction to obtain the coordinates of the reference detection box. For the first network header The reference detection box coordinates and the target image features are input into the local region feature extractor connected to the first network head for pooling. The resulting pooled features are input into the first network head for defect detection, and the defect detection box coordinates are output. For each of the remaining network headers The defect detection box coordinates output by the previous network head and the target image features are input to the local region feature extractor connected to the network head for pooling. The resulting pooled features are then input to the network head for defect detection, and the defect detection box coordinates are output. Specifically, the coordinates of the defect detection box output by the last network header are determined as the defect detection box information.

8. The defect detection method according to any one of claims 1 to 7, wherein, The defect detection box information includes defect detection box coordinates, detection box confidence probability, and target defect category. Determining the second detection result of the target workpiece based on the defect detection box information and the first detection result includes: When the first detection result indicates that the target workpiece has no defects, the preset probability threshold and preset area threshold corresponding to the target defect category are obtained; The defect area is determined based on the coordinates of the defect detection box, and a first relationship between the defect area and the preset area threshold is determined, as well as a second relationship between the confidence probability of the detection box and the preset confidence probability threshold. The second detection result is determined based on the first size relationship and the second size relationship.

9. The defect detection method according to claim 8 further includes: When the first detection result indicates that the target workpiece has a defect, the second detection result is determined to indicate that the target workpiece has a defect.

10. The defect detection method according to any one of claims 1 to 9, wherein, The step of comparing the image to be tested with the template image to obtain a difference image includes: Determine the transformation parameters between the image to be tested and the template image; According to the transformation parameters, the target workpiece displayed in the template image is aligned with the target workpiece displayed in the image to be tested to obtain the aligned image to be tested; The aligned image to be tested is compared with the template image to obtain an initial difference image; The initial difference image is filtered to obtain the difference image.

11. The defect detection method according to claim 10, wherein, Determining the transformation parameters between the image to be tested and the template image includes: Use a sliding window to determine the current window region in the image to be tested; Calculate the correlation coefficient between the image within the current window region and the image within the corresponding window region of the template image; Based on the correlation coefficients of each window region, the target region is determined in the image to be tested; Determine the transformation matrix required to transform the target region to the region aligned in the template image; The inverse of the transformation matrix is ​​used to determine the transformation parameters.

12. The defect detection method according to claim 1, wherein, The step of extracting target image features based on the image to be tested and the difference image includes: The image to be tested is concatenated with the difference image to obtain a concatenated image; The concatenated image is input into the defect detection model to extract the target image features.

13. The defect detection method according to any one of claims 1 to 12, wherein, The defect detection model is trained through the following steps: Acquire training images of the target workpiece, wherein the training images are labeled with corresponding defect category tags; The training image is compared with the template image to obtain a sample difference image; The training image and the sample difference image are input into the defect detection model to obtain sample detection box information in the training image, wherein the sample detection box information includes the sample defect category of the target workpiece; Determine the initial classification loss based on the sample defect category and the defect category label; The initial classification loss is adjusted according to a preset first adjustment parameter and a second adjustment parameter to obtain the target classification loss. The first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the defect category... The difficulty level of the category corresponding to the tag; The defect detection model is trained based on the target classification loss.

14. A defect detection device, comprising: The first processing module is used to acquire the image to be tested of the target workpiece and the template image of the target workpiece, and compare the image to be tested with the template image to obtain a difference image; The second processing module is used to input the image to be tested and the difference image into the defect detection model and perform the following processing: extracting target image features based on the image to be tested and the difference image; Based on the features of the target image, defect detection is performed on the image to be tested to obtain defect detection box information in the image to be tested; The third processing module is used to locate the defect region in the differential image and determine the first detection result of the target workpiece based on the defect region. The fourth processing module is used to determine the second detection result of the target workpiece based on the defect detection box information and the first detection result.

15. The defect detection device according to claim 14, wherein, The second processing module is used to extract a first image feature from the image to be tested; Extract the second image features from the difference image; The target image features are obtained by fusing the first image features and the second image features.

16. The defect detection device according to claim 14 or 15, wherein, The defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The second processing module is used to: input the target image features into the region proposal network for region extraction to obtain reference detection box coordinates; for the first network head, input the reference detection box coordinates and the target image features into the local region feature extractor connected to the first network head for pooling, input the obtained pooled features into the first network head for defect detection, and output the defect detection box coordinates; for each of the remaining network heads, input the defect detection box coordinates output by the previous network head and the target image features into the local region feature extractor connected to that network head for pooling, input the obtained pooled features into that network head for defect detection, and output the defect detection box coordinates; wherein, the defect detection box coordinates output by the last network head are determined as the defect detection box information.

17. The defect detection apparatus according to any one of claims 14 to 16, wherein, The defect detection box information includes defect detection box coordinates, detection box confidence probability, and target defect category. The fourth processing module is used to: when the first detection result indicates that the target workpiece does not have a defect, obtain a preset confidence probability threshold and a preset area threshold corresponding to the target defect category; determine the defect area based on the defect detection box coordinates; determine a first size relationship between the defect area and the preset area threshold, and a second size relationship between the detection box confidence probability and the preset confidence probability threshold; and determine the second detection result based on the first size relationship and the second size relationship.

18. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the defect detection method of any one of claims 1 to 13.

19. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the defect detection method according to any one of claims 1 to 13.

20. A computer program product comprising a computer program that, when executed by a processor, implements the defect detection method according to any one of claims 1 to 13.

Images