Defect detection method and apparatus, electronic device, storage medium and program product

Abstract

Disclosed in the embodiments of the present application are a defect detection method and apparatus, an electronic device, a storage medium and a program product. The method comprises: acquiring an image under test of a target workpiece and a template image of the target workpiece, and comparing the image under test with the template image so as to obtain a differential image; inputting the image under test and the differential image into a defect detection model, and executing the following processing: on the basis of the image under test and the differential image, extracting a target image feature and, on the basis of the target image feature, performing defect detection on the image under test to obtain defect detection box information of the image under test; positioning a defect area in the differential image and, according to the defect area, determining a first detection result of the target workpiece; and, according to the defect detection box information and the first detection result, determining a second detection result of the target workpiece.

Description

Defect detection method, device, electronic device, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 10, 2023, with application number 202311498351.5 and application name “Defect Detection Method, Device, Electronic Device and Storage Medium”. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a defect detection method, device, electronic device, storage medium and program product.

[0003] Background of the Invention

[0004] Currently, deep learning algorithms are commonly used to detect product defects. However, the accuracy of deep learning defect detection relies on a large number of manually labeled defect samples. As the number of defect types increases or the distinguishability of defect features decreases, the number of defect samples required increases, leading to high sample acquisition costs and low training efficiency. Furthermore, because deep learning algorithms generally use supervised training, a lack of training samples can reduce defect detection accuracy.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a defect detection method, apparatus, electronic device, storage medium, and program product, which can improve the accuracy of defect detection while reducing dependence on training samples.

[0007] In one aspect, an embodiment of the present application provides a defect detection method, comprising:

[0008] Acquire an image to be measured of a target workpiece and a template image of the target workpiece, and compare the image to be measured with the template image to obtain a differential image;

[0009] The image to be tested and the differential image are input into the defect detection model, and the following processing is performed:

[0010] Extracting target image features based on the image to be measured and the differential image;

[0011] Performing defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested;

[0012] Locating a defective area in the differential image, and determining a first detection result of the target workpiece based on the defective area; and

[0013] A second detection result of the target workpiece is determined according to the defect detection frame information and the first detection result.

[0014] On the other hand, an embodiment of the present application further provides a defect detection device, comprising:

[0015] a first processing module, configured to obtain an image to be measured of a target workpiece and a template image of the target workpiece, and compare the image to be measured with the template image to obtain a differential image;

[0016] a second processing module, configured to input the image to be tested and the differential image into a defect detection model and perform the following processing: extracting target image features based on the image to be tested and the differential image; performing defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested;

[0017] a third processing module, configured to locate a defective area in the differential image and determine a first detection result of the target workpiece according to the defective area;

[0018] A fourth processing module is configured to determine a second detection result of the target workpiece according to the defect detection frame information and the first detection result.

[0019] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned defect detection method when executing the computer program.

[0020] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned defect detection method.

[0021] In another aspect, embodiments of the present application further provide a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to implement the above-described defect detection method.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0024] FIG1 is a schematic diagram of an optional implementation environment provided in an embodiment of the present application;

[0025] FIG2 is a schematic diagram of an optional flow chart of a defect detection method provided in an embodiment of the present application;

[0026] FIG3 is a schematic diagram of comparing a test image with a template image according to an embodiment of the present application;

[0027] FIG4 is a schematic diagram of generating defect detection frame information of an image to be tested provided by an embodiment of the present application;

[0028] FIG5 is a schematic diagram of preprocessing of a test image and a differential image before feature extraction according to an embodiment of the present application;

[0029] FIG6 is a schematic diagram of a first image feature extraction process provided in an embodiment of the present application;

[0030] FIG7 is a schematic diagram of generating a convolution branch data stream during the process of extracting a first image feature according to an embodiment of the present application;

[0031] FIG8 is a schematic diagram of a first image feature extraction process provided by another embodiment of the present application;

[0032] FIG9 is a schematic diagram of a process for extracting first image features according to another embodiment of the present application;

[0033] FIG10 is a schematic diagram of extracting a second target convolution feature according to an embodiment of the present application;

[0034] FIG11 is a schematic diagram of a target image feature generation process provided by an embodiment of the present application;

[0035] FIG12 is a schematic diagram of a defect detection process provided in an embodiment of the present application;

[0036] FIG13 is a schematic diagram of a second detection result determination process provided in an embodiment of the present application;

[0037] FIG14 is a schematic diagram of the effect of morphological filtering provided in an embodiment of the present application;

[0038] FIG15 is a schematic diagram of a process for determining a correlation coefficient according to an embodiment of the present application;

[0039] FIG16 is a schematic diagram of a similarity transformation correspondence between a template image and an image to be tested provided in an embodiment of the present application;

[0040] FIG17 is a schematic diagram of the training process of the defect detection model provided in an embodiment of the present application;

[0041] FIG18 is a schematic diagram of a defect detection process provided by another embodiment of the present application;

[0042] FIG19 is a schematic diagram of a defect detection frame information generation process provided by another embodiment of the present application;

[0043] FIG20 is a schematic diagram of an optional overall flow chart of a defect detection method provided in an embodiment of the present application;

[0044] FIG21 is a schematic diagram of an optional overall flow chart of a defect detection method provided in an embodiment of the present application;

[0045] FIG22 is a schematic diagram of an optional structure of a defect detection device provided in an embodiment of the present application;

[0046] FIG23 is a partial structural block diagram of a terminal provided in an embodiment of the present application;

[0047] Figure 24 is a partial structural block diagram of the server provided in an embodiment of the present application.

[0048] Implementation Method

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] It should be noted that, in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the attribute information or attribute information set of the target object, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, a separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the relevant data of the target object necessary to enable the normal operation of the embodiment of the present application will be obtained.

[0051] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0052] Defect Detection usually refers to the detection of surface defects of objects. Machine vision detection technology is used in surface defect detection to detect defects such as spots, pits, scratches, color differences, and defects on the surface of the workpiece, and the defective parts can be marked.

[0053] The three primary colors (Red, Green, Blue, RGB) are a color model composed of the three primary colors red, green and blue. Various colors are obtained by changing the three color channels of red, green and blue and superimposing them on each other. In the fields of electronic displays, photography, computer graphics, etc., RGB is widely used to describe, control and display color images, and as the storage format and display format of photos after camera imaging.

[0054] Morphological image processing refers to a series of image processing techniques that process image shape features. The basic idea of ​​morphology is to use a special structure element to measure or extract the corresponding shape or feature in the input image for further image analysis and target recognition.

[0055] Currently, deep learning algorithms are commonly used to detect product defects. For example, in industrial manufacturing, artificial intelligence and deep learning technologies are often used to perform quality inspections on product appearance. When deep learning algorithms are used for defect detection, supervised training of deep learning network models is generally used to annotate defects in images of inspection workpieces. Abnormal areas are then identified through object detection or segmentation methods, and defects are then classified to achieve the desired defect detection results.

[0056] However, since defect annotation of images of quality inspection workpieces requires manual marking, the cost of sample production is high, and the more types of defects there are or the lower the distinguishability of the defect features, the larger the number of defect samples required. In addition, some samples with severe visual defects are difficult to obtain, resulting in the deep learning algorithm not having learned the features of similar defects and being unable to accurately distinguish the defects of quality inspection workpieces. Therefore, the accuracy of defect detection using deep learning algorithms depends on a large number of manually labeled defect samples, which are difficult and costly to obtain, and the accuracy of defect detection is reduced in the absence of samples.

[0057] In order to solve the above problems, the embodiments of the present application provide a defect detection method, device, electronic device, storage medium and program product, which can improve the accuracy of defect detection while reducing the dependence on training samples.

[0058] The method provided in the embodiments of the present application can be applied to different technical fields, including but not limited to artificial intelligence, cloud technology, computer vision technology, industrial automation and other scenarios.

[0059] 1 , which is a schematic diagram of an optional implementation environment provided by an embodiment of the present application, the implementation environment includes a terminal 101 and a server 102 , wherein the terminal 101 and the server 102 are connected via a communication network.

[0060] Terminal 101 may be, but is not limited to, a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal. Terminal 101 and server 102 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present application. Optionally, terminal 101 may acquire an image of the target workpiece to be measured and then transmit the image to server 102.

[0061] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0062] Alternatively, the server 102 may be a node server in the blockchain network. Optionally, the server 102 may pre-store a defect detection model, wherein the server 102 may also pre-store a template image of the target workpiece, or the server 102 may receive a template image of the target workpiece sent by the terminal 101.

[0063] When the server 102 obtains the image to be tested of the target workpiece and the template image of the target workpiece, the image to be tested can be compared with the template image to obtain a differential image, and then the image to be tested and the differential image can be respectively input into a pre-stored defect detection model to obtain defect detection frame information in the image to be tested. At the same time, the defect area can be located in the differential image, and the first detection result of the target workpiece (i.e., a reference detection result) can be determined based on the defect area, and then the second detection result of the target workpiece (i.e., the target detection result) can be determined based on the defect detection frame information and the reference defect detection information.

[0064] In this way, by combining the differential contrast processing with the deep learning algorithm, it is possible to use differential contrast to obtain features with high visual visibility, which can visually assist in defect detection, effectively improve the accuracy of defect detection, and reduce dependence on training samples.

[0065] Exemplarily, the terminal 101 can obtain the image to be tested and the template image of the target workpiece, and then send the image to be tested and the template image to the server 102. After the server 102 receives the image to be tested and the template image of the target workpiece, it can compare the image to be tested and the template image to obtain a differential image. The differential image can provide visual difference information of the target workpiece; then the image to be tested and the differential image are respectively input into the defect detection model to extract the target image features. By introducing difference information into the defect detection model, it helps to improve the accuracy of the model prediction and improve the reliability of the target image features.

[0066] Next, defect detection is performed on the image to be tested based on the target image features to obtain the defect detection frame information in the image to be tested. The differential image is used to highlight the features related to the defect area in the image to be tested, which helps to improve the defect detection model's ability to detect defect classifications that have not been learned due to the lack of training samples, and improve the accuracy of the defect detection frame information.

[0067] Afterwards, the defect area is located in the differential image, the first detection result of the target workpiece is determined based on the defect area, and the second detection result of the target workpiece is determined based on the first detection result and the defect detection frame information. In this way, the high visual visibility of the features obtained by comparison can be utilized to visually assist defect detection, effectively improve the accuracy of defect detection, and reduce dependence on training samples.

[0068] Referring to Figure 2, Figure 2 is an optional flow chart of the defect detection method provided in an embodiment of the present application. The defect detection method can be executed by the terminal, or by the server, or by the terminal and the server in cooperation. In the embodiment of the present application, the method is illustrated by taking the execution of the method by the server as an example. The defect detection method includes but is not limited to the following steps 201 to 204.

[0069] Step 201: obtaining an image to be measured of a target workpiece and a template image of the target workpiece, and comparing the image to be measured with the template image to obtain a differential image.

[0070] In one possible implementation, the image to be measured of the target workpiece can be acquired by a camera or sensor. The image to be measured can be a surface image, a cross-sectional image, an image obtained by infrared imaging, etc. of the target workpiece, depending on the object of defect detection and the detection requirements.

[0071] The template image of the target workpiece can be a reference image, i.e., an image of a known normal or defect-free workpiece, used for comparison with the image to be tested to detect differences or defects in the target workpiece. The template image can be acquired in advance during the design or manufacturing process of a workpiece of the same category as the target workpiece.

[0072] Specifically, for surface defect detection, a camera can be used to obtain a surface image of the target workpiece. The image to be tested can be an image of the front, side, or other angles of the target workpiece to comprehensively capture the features of the workpiece surface.

[0073] For internal defect detection or cross-section detection, a sensor (such as an X-ray sensor or an infrared sensor) can be used to obtain an internal structure image or a cross-section image of the target workpiece as the image to be tested.

[0074] Among them, defect detection can be applied to a variety of scenarios, such as steel plate defect detection, floor flaw detection, PCB board defect detection, film defect detection, lamp bead defect detection, metal bar end face detection, fabric wrinkle grade assessment, etc. The target workpieces are steel plates, floors, PCB boards, films, lamp beads, metal bars, fabrics, etc.

[0075] In one possible implementation, during the defect detection process, the image to be tested can be compared with the template image to find the differences and changes between the two, and a differential image can be generated, so that the possible defects or abnormalities on the target workpiece can be visually determined.

[0076] The color modes of the target workpiece image to be tested and the template image can be the same or different. Specifically, if both the target workpiece image to be tested and the template image are color RGB images, grayscale conversion needs to be performed on both images before the comparison process is performed. If the target workpiece image to be tested can be a color RGB image, and the template image can be a pre-stored grayscale binary image, grayscale conversion only needs to be performed on the target workpiece image before the comparison process is performed.

[0077] Specifically, a grayscale transformation process can be performed on the image to be tested and the template image of the target workpiece, respectively. This process includes calculating the grayscale difference between the current pixel and adjacent pixels in the image to obtain a pixel difference value, then comparing the pixel difference value with a preset grayscale difference threshold to obtain a binary pixel value for the current pixel, and then traversing each pixel in the image to obtain the binary pixel values ​​for all pixels. The adjacent pixels can refer to the pixels surrounding each pixel in the image, all pixels in the image, or pixels corresponding to the position of the current pixel in images of different scales.

[0078] The image to be tested and the template image are converted into binary images, that is, they are converted into grayscale images. The grayscale level of the image reflects the brightness value of each pixel in the image. The higher the grayscale level, the brighter the pixel. After obtaining the grayscale images corresponding to the image to be tested and the template image, the two can be compared. First, the two images are matched. By rotating, translating, stretching, and other morphological processing on the template image, the target workpiece displayed in the template image is aligned with the target workpiece displayed in the image to be tested. Then, the image to be tested and the adjusted template image are compared. That is, the corresponding pixel values ​​of the image to be tested and the adjusted template image are subtracted to reduce the similar parts of the two images and highlight the changes between the two images.

[0079] As shown in FIG3 , FIG3 is a schematic diagram of comparing a test image with a template image according to an embodiment of the present application. The absolute value obtained by subtracting the pixel value of each pixel point of the test image from the pixel value of the corresponding pixel point of the adjusted template image is used as the pixel value of each pixel point of the differential image. The calculation formula is as follows: Df(x,y)=abs[I(x,y)-T(x,y)] (1)

[0080] Where Df(x,y) represents the pixel value of the pixel with coordinates (x,y) in the differential image, I(x,y) represents the pixel value of the pixel with coordinates (x,y) in the image to be measured, and T(x,y) represents the pixel value of the pixel with coordinates (x,y) in the adjusted template image.

[0081] As shown in Figure 3, the difference image shows a difference region A, indicating a difference between the image to be tested and the adjusted template image. Comparing the adjusted template image and the image to be tested reveals that the adjusted template image shows a structural region corresponding to difference region A, while the image to be tested does not. This means that the reference workpiece, which is of the same type as the target workpiece, has a first structure in the location corresponding to difference region A, while the target workpiece lacks this first structure.

[0082] In one possible implementation, the template image can be pre-stored in a server or terminal. After acquiring an image of the target workpiece to be tested, target detection is performed on the target workpiece in the image to determine the workpiece category to which the target workpiece belongs. Based on the corresponding workpiece category, a template image of the same workpiece category is acquired and used as the template image of the target workpiece.

[0083] Step 202: Input the image to be tested and the differential image into the defect detection model and perform the following processing: extract the target image features based on the image to be tested and the differential image; perform defect detection on the image to be tested based on the target image features to obtain the defect detection frame information in the image to be tested.

[0084] In one possible implementation, the defect detection model can be a neural network model built based on a deep learning algorithm for automatically detecting and identifying workpiece defects. The defect detection model can learn and extract features from the input sample image, and predict and classify defects based on the extracted features.

[0085] Defect detection models are trained using supervised training, which requires a large, labeled dataset. The performance and accuracy of the defect detection model depend on the quality and quantity of the training data, making it difficult for the model to detect unlearned defect categories. Therefore, in addition to inputting the image to be tested into the defect detection model for defect detection, the defect detection model also inputs a difference image. The defect detection model then extracts target image features by combining the image to be tested and the difference image.

[0086] Since the differential image is obtained by comparing the image to be tested with the template image, it contains the difference information between the two. Therefore, by introducing the differential image to provide image difference information for the defect detection model, the defect detection model is assisted in extracting features from the image to be tested. It can more accurately extract the features of the corresponding area in the image to be tested based on the image difference information, which helps to improve the accuracy of subsequent defect detection.

[0087] In one possible implementation, the defect detection bounding box can be a rectangular box used to mark and locate the defect area in the image to be tested, and the defect detection frame information can refer to the position coordinates of the defect detection frame in the image to be tested, and can also include the confidence of the defect detection model output for the defect detection frame.

[0088] While the image under test may not clearly reflect defects, the differential image has high visual visibility and can highlight features of visually obvious defects in the image under test. The resulting target image features can help increase the defect detection model's sensitivity to visually obvious defects. Furthermore, the defect detection model is trained using supervised training. The target image features can supplement the difference information between the image under test and the template image, enabling the defect detection model to better detect defects it has not learned, thereby improving the generalization ability of the defect detection model.

[0089] Therefore, defect detection is performed on the image to be tested based on the target image features, so that the defect detection model can identify defects that are very small, visually inconspicuous or similar to the background, thereby improving the detection accuracy. This can help reduce the false detection rate of normal workpieces and the missed detection rate of defective workpieces. It can also assist the defect detection model in identifying defect features that have not been learned in the image to be tested, thereby improving the accuracy of generating defect detection frames, that is, improving the reliability of defect detection frame information.

[0090] In one possible implementation, when inputting the image to be tested and the differential image into the defect detection model and extracting the target image features, the first image features can be first extracted from the image to be tested; the second image features can be extracted from the differential image; and the first image features and the second image features can be fused to obtain the target image features.

[0091] In a possible implementation, feature fusion may include weighted fusion, feature concatenation, feature superposition, feature selection, and the like.

[0092] Specifically, a first weight can be assigned to the first image feature, a second weight can be assigned to the second image feature, and then the first and second image features can be weighted summed to obtain the target image feature. The first and second weights can be determined based on the importance of the features or obtained through model training.

[0093] Specifically, the first image feature and the second image feature can be concatenated in a preset order to form a larger vector, so that the information of the two features can be easily integrated together, and the dimension of the feature vector can be expanded, so that the defect detection model can simultaneously utilize the features of the image to be tested and the features of the differential image, thereby improving the performance of the defect detection model.

[0094] Specifically, the first image feature and the second image feature may be superimposed by addition and subtraction, multiplication, convolution or pooling operations to obtain the target image feature.

[0095] It is also possible to perform regional screening on the first image feature and the second image feature, select some key feature information from the first image feature and the second image feature respectively, and recombine them to form a target image feature.

[0096] The first image feature of the image to be tested captures the original information of the target workpiece, while the second image feature of the differential image captures the difference information between the target workpiece and the reference workpiece (the workpiece displayed in the template image of the target workpiece), that is, the possible defect information of the target workpiece. By fusing the first image feature and the second image feature, the information of the target workpiece can be represented more comprehensively and accurately, which helps to improve the accuracy of defect identification in subsequent defect detection steps.

[0097] Referring to Figure 4, Figure 4 is a schematic diagram of generating defect detection frame information for an image to be tested provided in an embodiment of the present application. First, the image to be tested (image A) and the differential image (differential image B) of the target workpiece are obtained. Then, using the feature extraction module in the defect detection model, feature extraction is performed on image A and differential image B, respectively, so that image feature A (i.e., the first image feature) of image A and differential image feature B (i.e., the second image feature) of differential image B can be obtained. Then, feature fusion is performed on image feature A and image feature B to obtain target image feature C. Then, defect detection can be performed on image A based on target image feature C to obtain defect detection frame information for image A.

[0098] The introduction of differential image feature B (i.e., the second image feature of the differential image) enables the defect detection model to focus more on the difference between the target workpiece and the reference workpiece, making defects that may have been hidden in background noise or surface texture easier to detect, thereby improving the sensitivity of the defect detection model to defects and thereby improving the accuracy of the defect detection frame information.

[0099] In addition, the feature extraction modules used to extract features of image A and differential image B can be different, and the two feature extraction modules can be trained using corresponding training samples. For example, if image A is a color image, the feature extraction module used to extract features of the image to be tested (image A) can use color images as training samples for feature extraction training, while if differential image B is a grayscale image, the feature extraction module used to extract features of differential image B can use grayscale images as training samples for feature extraction training. This allows for the extraction of more accurate image features for different image data.

[0100] In addition, the parameters of the feature extraction modules used to extract features of image A and differential image B can be the same, that is, they share the same feature extraction module, so that the feature extraction module of the defect detection model can learn two different information at the same time and more accurately capture the difference information between the two images.

[0101] Step 203: Locate the defect area in the differential image, and determine a first detection result of the target workpiece according to the defect area.

[0102] In one possible implementation, the pixel value of each pixel in the differential image can be compared with a preset pixel threshold. When the pixel value exceeds the preset pixel threshold, it can be considered that a defect exists at the location corresponding to the pixel. By counting the pixel values ​​exceeding the preset pixel threshold, the defective area can be located in the differential image.

[0103] In one possible implementation, the difference image can be a grayscale image, and connected domain analysis can be performed on the difference image to group adjacent pixels with the same pixel value into a connected domain. The adjacent pixels can be foreground pixels, that is, adjacent pixels with a pixel value of 1 are grouped into a connected domain. The location and size of the defect area can then be determined by calculating the bounding rectangle or minimum circumscribed circle of the connected domain. For example, the bounding rectangle of the connected domain can be calculated and used as the defect area.

[0104] In a possible implementation, shape features (such as aspect ratio, circularity, etc.) of the connected domain can be further calculated to verify whether the defect area meets the defect characteristics, so as to filter out defect areas that meet the defect characteristics and determine the first detection result.

[0105] In addition, before performing connected domain analysis on the differential image, morphological processing (such as opening or closing operations) can be performed on the differential image to reduce the impact of noise, and the connected domain can be filtered based on a preset connected domain threshold (such as the connected domain area or the connected domain perimeter, etc.) to exclude connected domains with too few pixels and improve the accuracy of the analysis.

[0106] In one possible implementation, after the defect area is determined, the first detection result of the target workpiece can be determined by judging whether the defect area meets the defect characteristic conditions, that is, the first detection result is used to indicate whether the target workpiece has defects based on the differential image.

[0107] Specifically, the defect characteristic conditions can include: the area, perimeter, shape of the defect region, or the difference between the pixel values ​​of pixels in the connected domain and those outside the connected domain, and whether this exceeds a pre-set defect threshold. If the defect region meets one or more of these defect characteristic conditions, the target workpiece can be considered defective, and the first inspection result is a failure. If the defect region does not meet these defect characteristic conditions, the target workpiece can be considered non-defective, and the first inspection result is a passing quality inspection.

[0108] Step 204: Determine a second detection result of the target workpiece according to the defect detection frame information and the first detection result.

[0109] In one possible implementation, the defect detection frame information includes areas of the image under test that are predicted by the defect detection model based on characteristics learned during training. The first detection result indicates the defect status of the target workpiece after comparing the template image with the image under test. This high visual visibility prevents missed defects that were not present during training but are visually apparent. It also provides a high recall rate for known defects, effectively improving detection accuracy.

[0110] Furthermore, computer vision technology is susceptible to interference from factors such as lighting conditions and workpiece surface texture, and some defects may be difficult to detect from a single perspective or dimension. However, combining multiple information sources can reduce the impact of these interference factors, improve detection stability and reliability, and enable more comprehensive and accurate defect detection. Therefore, by combining the defect detection frame information with the first detection result to determine the second detection result, false positives and missed detections can be effectively reduced, thereby improving the accuracy of target defect detection while reducing reliance on training samples.

[0111] In one possible implementation, it is possible to determine whether the target workpiece has passed the quality inspection by comparing the defect detection frame information with the preset defect area condition. For example, the preset defect condition may be that the area of ​​the defect detection frame indicated by the defect detection frame information is larger than the preset defect area, or that the defect detection frame indicated by the defect detection frame information contains an area that cannot be defective (such as a structural area that seriously affects the quality of the workpiece), etc. Therefore, when the defect detection frame information or the first test result indicates that the target workpiece has failed the quality inspection, it can be determined that the defects existing in the target workpiece cannot pass the quality inspection, that is, the second test result is that the quality inspection has failed; when the defect detection frame information and the first test result both indicate that the target workpiece has passed the quality inspection, it can be determined that the target workpiece has no defects or that the defects existing can pass the quality inspection, that is, the second test result is that the quality inspection has passed.

[0112] In one possible implementation, in the process of inputting the image to be tested and the differential image into the defect detection model respectively, the data of each channel in the differential image can be copied first until the number of channels of the copied differential image is equal to the number of channels of the image to be tested; the image to be tested and the copied differential image are input into the defect detection model respectively, wherein the image to be tested and the copied differential image share the same feature extraction module in the defect detection model.

[0113] Both the image to be tested and the differential image contain key information that helps identify defects. In order to enable the defect detection model to obtain more information and make more accurate predictions, both the image to be tested and the differential image can be input into the same feature extraction module for feature extraction. However, the feature extraction module requires all inputs to have the same dimension.

[0114] Referring to Figure 5, Figure 5 is a schematic diagram of preprocessing of a test image and a differential image prior to feature extraction, as provided in an embodiment of the present application. In this case, taking the differential image (Image 1) as a 3x3 grayscale image as an example, Image 1 is a single-channel image represented by a matrix H, where each element in the matrix H corresponds to a pixel in the image, and each element represents brightness. The value of each element can range from 0 to 255.

[0115] The image to be tested is a color image with multi-channel three-dimensional data. For example, the image to be tested (Image 2) is a 3x3 color RGB image with three channels. The data of channel R in the matrix represents the components of the red channel, the data of channel G represents the components of the green channel, and the data of channel B represents the components of the blue channel.

[0116] Therefore, the number of channels of the differential image can be made equal to the number of channels of the image to be measured by copying. As shown in Figure 5, the differential image (Image 1) is copied into an image with three-channel data (Image 3), where the data of channels H1 and H2 are the same as that of channel H.

[0117] The differential image and the image to be tested share the same feature extraction module in the defect detection model, which enables the defect detection model to process and learn two types of information with different representations in the same parameter space, implicitly providing the model with difference information about the target workpiece, and further improving the performance of the defect detection model. At the same time, since the feature extraction module reuses the same set of neural network weights, there is no need to process and learn a set of weights for each input separately, which can reduce the dependence on training samples and improve training efficiency.

[0118] In a possible implementation, in the process of extracting the first image feature of the image to be tested, multiple stages of convolution may be sequentially performed on the image to be tested, wherein:

[0119] Each stage includes at least one convolution branch, and each convolution branch is used to perform multiple convolutions;

[0120] When entering the next stage from the current stage, a new convolution branch is added, wherein the resolution of the new convolution branch is smaller than any existing convolution branch in the current stage;

[0121] The input feature of any convolution branch in the current stage is obtained by fusion of the output features of all convolution branches in the previous stage;

[0122] Then, the initial convolution features output by each convolution branch in the last stage are fused to obtain the first target convolution feature;

[0123] Next, the first target convolution feature is downsampled to obtain the first image feature.

[0124] First, by performing multiple stages of convolution on the image to be tested, multiple convolutions are performed during each stage. The first convolution can only extract low-level features such as lines or edges of the target object in the image (such as the structure of the target workpiece). As the number of convolution stages increases, deeper features can be effectively extracted, such as high-level features such as the texture and shape of the target object in the image. This helps the defect detection model gain a deep understanding of the specific content displayed in the image to more accurately detect missing parts. At the same time, the multiple convolutions can effectively reduce the spatial size of the image, reducing the amount of data required for subsequent calculations.

[0125] After obtaining high-level image features, a new convolution branch can be added. The resolution of the added convolution branch is smaller than the resolution of any existing convolution branch in the stage to which it belongs, and the input of any existing convolution branch in the stage is obtained by fusion of the outputs of all convolution branches in the previous stage.

[0126] By adding convolution branches with smaller resolution, more globally meaningful or abstract features can be detected on the already extracted features, gradually capturing higher-level semantic information. At the same time, the noise in the features can be reduced in the new convolution branches, thereby improving the reliability of the features.

[0127] Specifically, referring to FIG6 , FIG6 is a schematic diagram of a process for extracting first image features provided by an embodiment of the present application, wherein the process for extracting the first image features of the image to be tested adopts a parallel convolutional stream approach and has multiple stages. Starting from the first stage, data streams are gradually added from high resolution to low resolution, and the resolution of the parallel data stream in the subsequent stage is composed of the resolution of the previous stage and a lower resolution. As the number of stages increases, the number of different resolutions increases.

[0128] As shown in Figure 6, the new convolution branch is the data stream with the lowest resolution in the current stage. Its input is obtained by downsampling the features output by all convolution branches in the previous stage. When there are multiple convolution branches in the previous stage, the features output by all convolution branches can be downsampled to obtain features with the same resolution as the convolution branch to be added. These features with the same resolution are then fused (such as weighted fusion, cascade, superposition, etc.) as the input of the new convolution branch, thereby introducing a convolution branch with lower resolution to obtain image features with lower resolution.

[0129] Referring to FIG. 7 , FIG. 7 is a schematic diagram illustrating the generation of a convolution branch data stream during the extraction of the first image feature according to an embodiment of the present application. The extraction process shown in FIG. 7 corresponds to the extraction process shown in FIG. 6 . The process of extracting the first image feature of the image to be tested can be divided into four stages, wherein:

[0130] The first stage consists of a data stream N corresponding to a resolution 11 , which is the resolution of the original convolution branch that convolves the image to be tested (the first resolution).

[0131] The second stage includes two data streams corresponding to two resolutions, including a data stream N of the first resolution 21 and a data stream N lower than the first resolution (second resolution) 22 . The second resolution is smaller than the first resolution.

[0132] Similarly, the third stage contains three data streams N corresponding to three resolutions 31 、N 32 、N 33 , the fourth stage contains four data streams N corresponding to four resolutions 41 、N 42 、N 43 、N 44 , thereby introducing a convolution branch with lower resolution to obtain image features with lower resolution.

[0133] When entering a new convolution stage and adding a new convolution branch, the features output by the existing convolution branches in the previous stage are upsampled or downsampled accordingly and added to the input of the current convolution of the corresponding convolution branch. This can retain the original feature learning path of the previous stage, and at the same time enable features at different resolution levels to be fused and feature information of different resolutions to be exchanged.

[0134] Specifically, referring to Figure 8, which is a schematic diagram of a first image feature extraction process provided by another embodiment of the present application, if the resolution of the current convolution branch to be sampled is smaller than the resolution of the target convolution branch, the output of the current convolution branch to be sampled at the previous stage is upsampled to obtain an upsampled feature with the same resolution as the resolution of the target convolution branch, and then the upsampled feature is used as the input of the target convolution branch at the current stage, so that the target convolution branch performs convolution on the upsampled feature.

[0135] If the resolution of the current convolution branch to be sampled is greater than the resolution of the target convolution branch, the output of the current convolution branch in the previous stage is downsampled to obtain a downsampled feature with the same resolution as the target convolution branch, and then the downsampled feature is used as the input of the target convolution branch in the current stage, so that the target convolution branch convolves the downsampled feature.

[0136] In the case where multiple downsampling is required, multiple downsampling methods can be used to sample to the required resolution.

[0137] If the resolution of the current convolution branch to be sampled is equal to the resolution of the target convolution branch, the output of the current convolution branch to be sampled in the previous stage is retained as the input of the convolution branch (target convolution branch) in the current stage, or the output of the current convolution branch to be sampled in the previous stage is resampled as the input of the convolution branch (target convolution branch) in the current stage.

[0138] As shown in FIG8 , the first resolution D1 is greater than the second resolution D2 , and the second resolution D2 is greater than the third resolution D3 .

[0139] If the resolution of the target convolution branch is the first resolution D1, the convolution branch L1 corresponding to the first resolution can directly add the convolution features output by itself in the previous stage to the input of the current stage, while the convolution branch L2 corresponding to the second resolution D2 and the convolution branch L3 corresponding to the third resolution need to upsample the convolution features output by themselves in the previous stage to the first resolution D1 and add them to the input of the convolution branch L1 in the current stage.

[0140] If the resolution of the target convolution branch is the second resolution D2, the convolution branch L1 corresponding to the first resolution needs to downsample the convolution features output by itself in the previous stage to the second resolution D2, and add them to the input of the convolution branch L2 in the current stage. The convolution branch L2 corresponding to the second resolution can directly add the convolution features output by itself in the previous stage to the input of the current stage. The convolution branch L3 corresponding to the third resolution needs to upsample the convolution features output by itself in the previous stage to the second resolution D2, and add them to the input of the convolution branch L2 in the current stage.

[0141] If the resolution of the target convolution branch is the third resolution D3, the convolution branch L1 corresponding to the first resolution and the convolution branch L2 corresponding to the second resolution D2 need to downsample the convolution features output by themselves in the previous stage to the third resolution D3 and add them to the input of the convolution branch L3 in the current stage, while the convolution branch L3 corresponding to the third resolution can directly add the convolution features output by themselves in the previous stage to the input of the current stage.

[0142] Therefore, when convolution starts at each stage, the features of the input data of each current convolution branch are first fused to reduce the amount of data processing.

[0143] By repeatedly exchanging information in parallel data streams to repeat multi-resolution fusion, high-resolution and low-resolution are connected in parallel and advanced synchronously, and high-resolution features and low-resolution features continuously exchange information. The extracted high-resolution features can make the first image features more spatially accurate in the subsequent feature fusion step, and the extracted low-resolution features can make the first image features more semantically sufficient in the subsequent feature fusion step.

[0144] In one possible implementation, in the process of fusing the initial convolution features output by each convolution branch in the last stage to obtain the first target convolution feature, multiple initial convolution features can be fused first to obtain the fused feature; for each of the multiple convolution branches in the last stage, a convolution with the same resolution as the convolution branch is performed on the fused feature to obtain the second target convolution feature of the convolution branch; and multiple of the second target convolution features are fused to obtain the first target convolution feature.

[0145] The second target convolution features of multiple different resolutions are fused to obtain the first target convolution feature, so that the first target convolution feature can contain more comprehensive and rich feature information.

[0146] Next, the first target convolution feature is downsampled and further compressed to obtain a feature representation, and more abstract and globally semantic features are extracted to obtain the first image feature of the image to be tested.

[0147] Specifically, referring to Figure 9, Figure 9 is a schematic diagram of the process of extracting the first image features provided by another embodiment of the present application. As shown in Figure 9, after obtaining the initial convolution features output by all convolution branches (including convolution branch L1, convolution branch L2, convolution branch L3 and convolution branch L4) in the last stage, the initial convolution features output by each convolution branch are convolved based on the resolution of the convolution branch itself, so that each convolution branch can obtain a second target convolution feature with the same resolution as its own.

[0148] Then, all the second target convolution features can be spliced ​​to form the first target convolution feature. At this time, the number of channels of the first target convolution feature is much larger than the number of channels of the image to be tested.

[0149] Then, the first target convolution feature is downsampled, the target convolution feature is compressed, and the dimension of the first target convolution feature is reduced so that the resolution of the first image feature is equal to the minimum resolution of the output features in all convolution branches.

[0150] In one possible implementation, in the process of performing multiple stages of convolution on the image to be tested, when the minimum resolution of the current stage meets the preset resolution threshold, the addition of new convolution branches is stopped, and the current stage is used as the last stage of the convolution process, that is, each convolution branch performs the last convolution.

[0151] Since each introduction of a new convolution branch will increase the number of convolution operations, it is easy to make the model too complicated and reduce the training efficiency. In addition, too low a resolution may lose a lot of useful detail information, affecting the quality of the first image feature obtained by subsequent feature fusion. Therefore, by judging whether the minimum resolution of the current stage meets the preset resolution threshold, it is determined whether the current stage is the last stage of the convolution process.

[0152] When the current stage is the last stage, the output of the current convolution branch in the last stage (i.e., the initial convolution feature) is upsampled or downsampled accordingly according to the resolution relationship between the current convolution branch and the remaining convolution branches, and the convolution feature with a resolution corresponding to the resolution of the remaining convolution branches is obtained, and the second target convolution feature of each convolution branch is obtained.

[0153] Referring to Figure 10, Figure 10 is a schematic diagram of extracting the second target convolution feature provided by an embodiment of the present application. As shown in Figure 10, when the resolution corresponding to the new convolution branch (convolution branch X) introduced in the Mth stage meets the preset resolution threshold, the Mth stage can be determined to be the last stage of feature extraction. Therefore, the initial convolution features output by all convolution branches in the Mth stage (the last stage) can be obtained, and multiple initial convolution features can be fused, and then the fused convolution features are respectively convolved with the same resolution as each convolution branch to obtain the second target convolution feature of the corresponding resolution.

[0154] Specifically, in the process of fusing multiple initial convolution features, each initial convolution feature can be upsampled or downsampled according to the resolution size relationship between each convolution branch and the remaining convolution branches, and added to the feature set to be fused of the convolution branch of the corresponding resolution in the Mth stage, and then all the initial convolution features in the feature set to be fused are convolved with the corresponding resolution to generate a second target convolution feature with the same resolution as each convolution branch.

[0155] In one possible implementation, in the process of fusing the first image feature and the second image feature to obtain the target image feature, the feature value of each pixel in the first image feature can be cascaded with the feature value of the corresponding pixel in the second image feature to obtain the third image feature; then, the third image feature is reduced in dimension to obtain the fourth image feature; then, the fourth image feature is activated to obtain the target image feature.

[0156] Referring to Figure 11, Figure 11 is a schematic diagram of the target image feature generation process provided by an embodiment of the present application. The differential image is a grayscale image, that is, a single-channel image, and the image to be tested is a color RGB image, that is, a three-channel image. In order to be able to reuse the feature extraction module and accelerate the training convergence, the single-channel data of the differential image can be copied first to form an image of three-channel data, so that the number of channels of the differential image is equal to the number of channels of the image to be tested. Then, the differential image of the three-channel data and the image to be tested are respectively input into the same feature extraction module in the defect detection model to obtain the first image feature and the second image feature, respectively, wherein the dimension of the first image feature is the same as the dimension of the second image feature, and the dimension is both 256.

[0157] After cascading the first image feature and the second image feature, the third image feature is obtained. The cascade operation is equivalent to splicing the two feature tensors along the channel dimension (or feature dimension), which can fuse and retain the information contained in the first image feature and the second image feature.

[0158] Since the dimension corresponding to the fused third image feature is 256*2, the third image feature contains redundant information. Therefore, a 1*1 convolution kernel can be used to perform channel dimensionality reduction on the third image feature to obtain a fourth image feature with a dimension of 256, so as to reduce unnecessary feature dimensions, compress feature representation, and reduce the computational overhead of subsequent processing. At the same time, mapping high-dimensional features to lower-dimensional space can effectively extract higher-level feature representations from the third image feature, which is conducive to extracting more global and abstract information and improving the quality and reliability of the fourth image feature.

[0159] Then, the fourth image feature is activated by using an activation function to obtain the target image feature, that is, the number of channels of the fourth image feature and the target image feature is the same as the number of channels of the image to be tested.

[0160] In one possible implementation, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. In the process of performing defect detection on an image to be tested based on target image features and obtaining defect detection frame information in the image to be tested, the target image features can first be input into the region proposal network for region extraction to obtain reference detection frame coordinates.

[0161] For the first network head, inputting the reference detection frame coordinates and the target image features into a local area feature extractor connected to the first network head for pooling, inputting the obtained pooled features into the first network head for defect detection, and outputting the defect detection frame coordinates;

[0162] For each of the remaining network heads, the coordinates of the defect detection frame output by the previous network head and the target image features are input into the local area feature extractor connected to the network head for pooling, the obtained pooled features are input into the network head for defect detection, and the coordinates of the defect detection frame are output;

[0163] The coordinates of the defect detection frame output by the last network head are determined as the defect detection frame information.

[0164] Specifically, the pooling process can adopt the Region of Interest Pooling (ROIPooling) technology, which can extract feature regions of a specific size, extract convolutional feature maps corresponding to regions of interest (ROI) of different sizes from the target image features according to the feature mapping relationship, and pool them to form feature maps of the same dimension.

[0165] The Region Proposal Network (RPN) is a fully convolutional network that takes an image of any size as input and outputs a set of rectangular object proposals, each with an objectness score. Therefore, the RPN can be used to predict the boundaries of defect detection boxes and the confidence level of the defect detection boxes based on the target image features.

[0166] Specifically, the target image features are input into the region proposal network, the possible defect areas in the target image features are extracted, and the coordinates of the reference detection box are output.

[0167] After obtaining the preliminary reference detection frame coordinates using the region proposal network, the network head in the subsequent stage can use the preliminary reference detection frame coordinates to optimize and obtain more detailed and accurate defect detection frame coordinates.

[0168] All network heads other than the first one can receive the output of the previous network head. The multi-cascade approach enables the network head to continuously learn and extract higher-level feature representations, further optimize and refine the defect detection frame, and improve the accuracy of the defect detection frame.

[0169] Referring to Figure 12, which is a schematic diagram of a defect detection process provided by an embodiment of the present application, the defect detection model includes multiple cascaded network heads. As shown in Figure 12, the defect detection model may include network head H1, network head H2, and network head H3, as well as a local area feature extractor, where network head H1 is the first network head and network head H3 is the last network head.

[0170] First, the target image feature (i.e., image feature P) is input into the region proposal network RPN for region extraction to obtain the reference coordinate frame coordinate B0.

[0171] Next, the reference detection frame coordinates B0 and image features P are input into the first local region feature extractor for region-of-interest pooling. Pooled features are extracted and fed into the first network head H1. Network head H1 adjusts the position and size of the defect detection frame based on the reference detection frame coordinates B0 and outputs the defect detection frame coordinates B1. Network head H1 also classifies the defect type and outputs a classification result C1 for the defect detection frame coordinates B1. Classification result C1 indicates the defect type corresponding to the defect detection frame coordinates B1.

[0172] Then, the defect detection frame coordinates B1 and the image features P are input into the second local area feature extractor for pooling of the region of interest, the pooling features of this stage are extracted, and sent to the next network head H2 for defect detection, the position and size of the defect detection frame are further adjusted, and the new defect detection frame coordinates B2 are output. At the same time, the network head H2 can also output the classification result C2 for indicating the defect type of the defect detection frame coordinates B2.

[0173] Next, the defect detection frame coordinates B3 and image features P are input into the third local region feature extractor for region-of-interest pooling. After extracting the new pooled features, they are input into the final network head H3. Network head H3 performs a final defect detection based on the pooled features, adjusts the position and size of the defect detection frame again, outputs the new defect detection frame coordinates B3, and uses the defect detection frame coordinates B3 as the defect detection frame information for the image under test. Furthermore, network head H3 also detects the classification result C3 corresponding to the defect detection frame coordinates B3, so that the defect detection frame information for the image under test can also include the classification result of the defect type corresponding to the defect detection frame.

[0174] Since the detection results of each stage will affect the effect of defect detection in the next stage, the iterative approach enables the model to gradually improve the detection results. That is, the reference defect detection box is first roughly located through the region proposal network RPN, and then the position and size of the defect detection box are continuously refined through multiple cascaded network heads to improve the detection accuracy.

[0175] In one possible implementation, the defect detection frame information includes the defect detection frame coordinates, the detection frame confidence probability, and the target defect category. In the process of determining the second detection result of the target workpiece based on the defect detection frame information and the first detection result, when the first detection result indicates that there is no defect in the target workpiece, the preset confidence probability threshold and the preset area threshold corresponding to the target defect category are obtained; then, the defect area is determined according to the defect detection frame coordinates, and a first size relationship between the defect area and the preset area threshold, as well as a second size relationship between the detection frame confidence probability and the preset confidence probability threshold are determined; then, based on the first size relationship and the second size relationship, the second detection result of the target workpiece is determined.

[0176] When the first detection result indicates that there are no defects in the target workpiece, it means that based on the differential image, it is determined that there are no visually obvious defects in the target workpiece. Therefore, the preset confidence probability threshold and the preset area threshold can be used to filter the defect detection box information output by the defect detection model to further determine whether there are defects in the target workpiece and reduce the occurrence of false alarms and missed alarms.

[0177] When outputting the coordinates of the defect detection frame, the defect detection model also outputs the target defect category (i.e., the classification result in Figure 12) and the detection frame confidence probability corresponding to each defect detection frame coordinate.

[0178] Target workpieces can have multiple defect categories, such as dirt, discoloration, scratches, damage, and dents. Quality inspection standards for different defect categories vary. Therefore, it's necessary to obtain a preset confidence probability threshold and a preset area threshold, i.e., the quality inspection standard, based on the target defect category corresponding to each defect detection frame. By comparing the first relationship between the defect area of ​​the defect detection frame and the preset area threshold, and the second relationship between the detection confidence probability and the preset confidence probability threshold, it is determined whether the defect detection frame contains defects that fail quality inspection.

[0179] The first size relationship indicates whether the defect area exceeds a preset area threshold, which can reflect the quality of the target workpiece. If the defect area is greater than or equal to the preset area threshold, it can be considered that the quality of the target workpiece does not meet the standard and the target workpiece has a defect.

[0180] The second magnitude relationship is used to indicate the probability of a defect detection frame, reflecting the likelihood of the target workpiece containing the defect. If the second magnitude relationship indicates that the detection frame confidence probability is greater than a preset confidence probability threshold, it can be assumed that the target workpiece has a defect in the location corresponding to the defect detection frame.

[0181] The second detection result of the target workpiece is determined based on the first size relationship and the second size relationship, and the judgment logic can be adjusted according to the actual quality inspection situation. For example, when the first size relationship is that the defect area is greater than or equal to the preset area threshold, or the second size relationship is that the detection frame confidence probability is greater than the preset confidence probability threshold, then it can be determined that the second detection result of the target workpiece is that the target workpiece has a defect and cannot pass the quality inspection; for example, when the first size relationship is that the defect area is greater than or equal to the preset area threshold, and the second size relationship is that the detection frame confidence probability is greater than the preset confidence probability threshold, then it can be determined that the second detection result of the target workpiece is that the target workpiece has a defect.

[0182] Specifically, as shown in Figure 13, Figure 13 is a schematic diagram of the second detection result determination process provided by an embodiment of the present application. When the first detection result indicates that there is no defect in the target workpiece, the defect detection frame information output by the defect detection model is obtained, specifically including the defect detection frame coordinates K1, K2, and K3. Among them, the detection frame confidence probability P1 corresponding to the defect detection frame coordinate K1 is 0.8, and the corresponding target defect category is dirt; the detection frame confidence probability P2 corresponding to the defect detection frame coordinate K2 is 0.3, and the corresponding target defect category is scratch; the detection frame confidence probability P3 corresponding to the defect detection frame coordinate K3 is 0.5, and the corresponding target defect category is concave.

[0183] Next, the preset confidence probability threshold and preset area threshold corresponding to the defect categories of dirt, scratches, and dents are obtained respectively. Specifically, the preset contamination confidence probability threshold corresponding to dirt is 0.7, and the preset contamination area threshold is 15 cm 2 The preset scratch confidence probability threshold corresponding to the scratch is 0.4, and the preset scratch area is 3cm 2 The preset concave confidence probability threshold corresponding to the concave is 0.7, and the preset concave area is 10cm 2 .

[0184] Then, the corresponding defect area is determined according to the coordinates of each defect detection frame. Specifically, the dirt defect area S1 calculated from the defect detection frame coordinate K1 is 20cm 2 The scratch defect area S2 calculated by the defect detection frame K2 is 3.5cm 2 , and the concave defect area S3 calculated by the defect detection frame coordinate K3 is 2cm 2 .

[0185] By comparing the defect area of ​​each defect category with the preset area threshold, as well as the detection frame confidence probability and the preset confidence probability threshold, it can be found that the first size relationship for the dirt category is that the dirt defect area is greater than the preset dirt area threshold, and the second size relationship for the dirt category is that the dirt detection frame confidence probability is greater than the preset dirt confidence probability. Therefore, the defect delineated by the defect detection frame coordinate K1 is considered to be a defect that cannot pass quality inspection. The first size relationship for the concave category is that the concave defect area is less than the preset concave area threshold, and the second size relationship for the scratch category is that the scratch detection frame confidence probability is less than the preset scratch confidence probability. Therefore, the defects delineated by the defect detection frame coordinates K2 and K3 can be considered to be defects that can pass quality inspection. However, since the target workpiece contains a defect that cannot pass quality inspection, namely the defect delineated by the defect detection frame coordinate K1, it can be determined that the second inspection result of the target workpiece is that the target workpiece has a defect and cannot pass quality inspection.

[0186] In one possible implementation, when the first inspection result indicates a defect in the target workpiece, the second inspection result of the target workpiece is determined to indicate a defect in the target workpiece. If the first inspection result indicates a defect in the target workpiece, this indicates that the target workpiece already has a visually obvious defect, and no further determination is required. The second inspection result of the target workpiece can be directly determined to indicate a defect in the target workpiece, thereby improving the efficiency of defect detection.

[0187] The target workpiece can be inspected for defects based on the comparison method and the model prediction method respectively.

[0188] The target workpiece first obtains a first detection result based on a comparison method. If the first detection result shows that the target workpiece has a defect, it can be directly concluded that the second detection result shows that the target workpiece has a defect, without considering the defect detection box information obtained by the model prediction method.

[0189] If the reference defect result is that there is no defect in the target workpiece, it is necessary to obtain the defect detection frame information output by the defect detection model, and then compare the defect detection frame coordinates, detection frame confidence probability and target defect category in the defect detection frame information with the preset confidence probability threshold and preset area threshold corresponding to the target defect category to determine the target reference defect result.

[0190] In one possible implementation, in the process of comparing the image to be tested with the template image to obtain a differential image, the transformation parameters between the image to be tested and the template image can be first determined; according to the transformation parameters, the target workpiece displayed in the template image and the target workpiece displayed in the image to be tested are aligned to obtain an aligned image to be tested; the aligned image to be tested is compared with the template image to obtain an initial differential image; and the initial differential image is filtered to obtain the differential image.

[0191] Since the image of the target workpiece to be tested is usually obtained by taking a picture with a camera, the shooting position or placement direction of each target workpiece is prone to deviation. Moreover, the image to be tested and the template image are acquired under different conditions such as different times and different viewing angles. As a result, the target workpiece displayed in the image to be tested is difficult to align with the target workpiece displayed in the template image, thus affecting the contrast effect.

[0192] Therefore, by aligning the image to be tested with the template image, the target workpiece displayed in the image to be tested and the target workpiece displayed in the template image can be aligned, thereby improving the contrast effect and the accuracy of the initial differential image.

[0193] Specifically, the transformation parameters between the image to be tested and the template image are first determined, and the image to be tested and the template image are aligned in the same coordinate space so that pixels at the same coordinate position correspond to the actual same position, thereby ensuring that the content of the same area is compared during the comparison process.

[0194] Determining the transformation parameters and performing image alignment can achieve spatial consistency between the image to be measured and the template image, help to effectively refine the information brought by the two images, and eliminate the noise caused by the framing difference, thereby helping to obtain a more accurate initial differential image.

[0195] Referring to Figure 14, which is a schematic diagram illustrating the effects of morphological filtering provided by an embodiment of the present application, even if the image to be tested and the template image are aligned, it is inevitable that there will be misaligned portions in the edge regions of both, which can easily lead to the appearance of noise and interfere with the generation of the differential image. Therefore, morphological image processing and filtering (such as dilation, erosion, filtering, smoothing, etc.) can be performed on the initial differential image to remove edge noise.

[0196] As shown in sub-image (a) in FIG14 , an erosion operation can be performed on the initial differential image to erode or shrink the edges in the initial differential image, thereby obtaining the eroded effect shown in sub-image (b) in FIG14 . That is, each pixel f(x, y) in the initial differential image f is replaced with the minimum value in set B (adjacent pixels), and the erosion operation on the initial differential image f is performed using set B. The specific formula is as follows:

[0197] In another embodiment, the edges in the initial differential image may be dilated (or expanded) to obtain the dilated effect shown in sub-image (c) of FIG16 . That is, each pixel f(x, y) in the initial differential image f is replaced with the maximum value in a set C (adjacent pixels), and the initial differential image f is dilated using the set C. The specific formula is as follows:

[0198] After morphological filtering, the noise at the edge of the initial differential image can be removed to obtain a differential image. The differential image and the image to be tested are input into the defect detection model, which can assist the model in feature extraction and defect detection. Thus, the model can accurately locate the defect area image as shown in sub-image (d) in Figure 14. The defect area image is part of the image information in the defect detection box information output by the model.

[0199] In one possible implementation, during the process of determining the transformation parameters between the image to be tested and the template image, a sliding window may be used to determine a current window region in the image to be tested; calculate a correlation coefficient between the image within the current window region and the image within the corresponding window region in the template image; determine a target region in the image to be tested based on the correlation coefficients of the respective window regions; determine a transformation matrix required to transform the target region to an aligned region in the template image; and determine the transformation parameters using the inverse matrix of the transformation matrix.

[0200] Referring to Figure 15 , which is a schematic diagram of the correlation coefficient determination process provided in an embodiment of the present application, the sliding window method involves sliding the image to be tested according to the size of the window and calculating the correlation coefficient between the template image and the corresponding window area in the image to be tested each time. This method measures the degree of similarity between the two and allows the region in the image to be tested that is most similar to the template image. The target region is the window area with the largest correlation coefficient.

[0201] After knowing the target area, the transformation matrix can be determined. The transformation matrix describes the geometric transformations such as displacement, rotation, or distortion required to transform the target area in the image to be tested into an area aligned with the template image. The transformation matrix M can be shown as follows:

[0202] Among them, (t x , t y ) represents the displacement vector. The rotation angle θ is added to the displacement vector to realize the rigid body transformation. The scaling transformation parameter s is added to the displacement and rotation to realize the similarity transformation. The similarity transformation changes the distance between the points before and after the transformation, but keeps the angle unchanged.

[0203] As shown in Figure 16, Figure 16 is a schematic diagram of the similarity transformation correspondence between the template image and the image to be tested provided by the embodiment of the present application. After determining the transformation matrix M, it is also necessary to calculate the inverse matrix M of the transformation matrix M. inv , the inverse matrix M inv It can be applied to convert the template image into the target area in the image to be measured, that is, the target workpiece displayed in the template image is transformed to the position of the target workpiece displayed in the target area in the image to be measured, thereby realizing workpiece alignment.

[0204] The correlation coefficient between the template image and the corresponding area in the image to be tested can be calculated based on the correlation between the relative values ​​of the pixel values ​​of the template image and its pixel mean, and the relative values ​​of the pixel values ​​of the corresponding area in the image to be tested and its pixel mean. The larger the correlation coefficient, the closer the match between the image to be tested and the corresponding area in the template image. The calculation formula of the correlation coefficient is as follows:

[0205] Where T′(x′,y′) represents the relative value of the pixel value at the (x′,y′) coordinate in the template image and the pixel mean of the template image, and I′(x+x′,y+y′) represents the relative value of the pixel value at the (x+x′,y+y′) coordinate in the corresponding window area in the image to be tested and the pixel mean of the window area.

[0206] In one possible implementation, the template image and the image to be tested can be converted into grayscale respectively, and then the difference, mean absolute difference, sum of absolute errors, mean sum of squared errors, normalized product, etc. can be calculated as correlation coefficients based on the grayscale values ​​of the sliding window area in the image to be tested and the template image to determine the target area.

[0207] In one possible implementation, a deep learning registration algorithm can be used to perform image matching between the sliding window area in the image to be tested and the template image, such as a feature matching algorithm based on a graph convolutional neural network (SuperGlue) or a self-supervised feature point detection algorithm based on deep learning (SuperPoint).

[0208] In one possible implementation, referring to FIG. 17 , which is a schematic diagram of a training process for a defect detection model provided in an embodiment of the present application, the defect detection model may be trained by following steps 1701 to 1705:

[0209] Step 1701: Obtain a training image of the target workpiece, compare the training image with the template image, and obtain a sample difference image, wherein the training image is annotated with a corresponding defect category label;

[0210] Step 1702: Input the training image and the sample difference image into the defect detection model to obtain sample detection frame information in the training image, wherein the sample detection frame information includes the sample defect category of the target workpiece;

[0211] Step 1703: Determine the initial classification loss based on the sample defect category and the defect category label;

[0212] Step 1704: Adjust the initial classification loss according to a preset first adjustment parameter and a second adjustment parameter to obtain a target classification loss, wherein the first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the classification difficulty corresponding to the defect category label;

[0213] Step 1705: Train a defect detection model based on the target classification loss.

[0214] Specifically, by comparing the training image with the template image, the defect characteristics of the target workpiece can be better learned. The sample differential image is helpful in highlighting the difference between the defect area and the normal area. The training image and the sample differential image are then input into the defect detection model. The sample differential image provides differential information for the model, assists model learning and training, and improves the generalization ability of the model.

[0215] The initial classification loss can be determined based on the cross entropy loss function using the sample defect category and the defect category label.

[0216] Next, the first adjustment parameter and the second adjustment parameter are preset. The first adjustment parameter and the second adjustment parameter can be used to adjust the initial classification loss to obtain the target loss function. Specifically, the target loss function can be shown as the following formula:

[0217] Among them, x represents the prediction score vector output by the defect detection model, class represents the defect category label, x[class] represents the original prediction score of the defect detection model for the current output defect belonging to a specific category, α class is represented as the first adjustment parameter, γ is represented as the second adjustment parameter, and It represents the cross entropy operation process, which converts the original output of the model into a probability form to obtain the predicted probability that the current output defect belongs to a specific category. The larger the predicted probability, the closer the current output defect is to the specific category and the more accurate the classification.

[0218] Compared with the cross entropy loss function, the target loss function adds a modulation factor For accurately classified samples, the predicted probability approaches 1 and the modulation factor approaches 0, while for inaccurately classified samples, the modulation factor approaches 0. Therefore, for the cross-entropy loss function, the target loss function does not change for inaccurately classified samples, while the loss decreases for accurately classified samples. Therefore, overall, this is equivalent to increasing the weight of inaccurately classified samples in the target loss function. At the same time, the output predicted probability can reflect the difficulty of the model classification. The larger the predicted probability, the higher the confidence of the model classification, which means the sample is easier to classify; while the smaller the predicted probability, the lower the confidence of the model classification, which means the sample is more difficult to classify.

[0219] Therefore, the target loss function can increase the weight of samples that are difficult to distinguish in the target loss function, making the target loss function tend to be difficult to distinguish samples, which helps to improve the accuracy of classification of difficult samples.

[0220] Since training images of defect-free target workpieces are easy to obtain, while target workpieces with defects are difficult to obtain, and as the number of defect types increases, the number of defect samples required increases, the first adjustment parameter can be used to suppress the imbalance in the number of positive and negative samples, and the second adjustment parameter can be used to control the imbalance in the number of simple / difficult to distinguish samples, where the positive sample can be represented as a sample image of a target workpiece with defects, and the negative sample can be represented as a sample image of a target workpiece without defects.

[0221] According to the above formula (5), it can be seen that the first adjustment parameter is used to adjust the loss ratio between positive and negative samples, while the second adjustment parameter can be used to adjust the loss ratio between difficult and easy samples. This is used to solve the class imbalance problem in target detection and reduce the weight of easy-to-classify samples, so that the model can focus more on difficult samples during training.

[0222] In one possible implementation, the second adjustment parameter may include a first decoupling factor and a second decoupling factor, wherein the first decoupling factor is represented as a basic value for adjusting the loss ratio between samples of different classification difficulty levels, and the second decoupling factor is represented as a variable parameter related to the imbalance of samples in each category.

[0223] Specifically, referring to the second adjustment parameter γ in formula (5), Among them, γ b Expressed as the first decoupling factor, is represented as the second decoupling factor, j represents the j-th category sample, g j represents the training balance coefficient of the j-th category sample, g j The larger the value, the more balanced the training of the j-th class sample is. jThe smaller the value, the more unbalanced the training of the category. s represents the scaling factor that determines the upper limit of the second adjustment parameter in the target loss function, which is used to assist in adjusting the imbalance in the number of simple / difficult-to-distinguish samples.

[0224] Therefore, by decoupling the second adjustment parameter to form the first decoupling factor and the second decoupling factor, the first decoupling factor and the second decoupling factor can be used to independently handle the imbalance of samples in each category, thereby improving the performance of the model. Specifically, taking the total number of sample categories as C as an example, the target loss function combining the first adjustment parameter, the first decoupling factor and the second decoupling factor can be expressed as follows:

[0225] In one possible implementation, during the training of the defect detection model, the sample detection frame information includes the defect detection frame and the corresponding detection frame confidence. After obtaining the sample frame information, a non-maximum suppression process can be performed according to the detection frame confidence, and the defect detection frames can be sorted according to the detection frame confidence corresponding to each defect detection frame. The defect detection frames of difficult samples (low detection frame confidence) are input into the next stage of training for forward propagation and back propagation, which enables the model to obtain more targeted information during supervised training and improve the speed and quality of training.

[0226] Specifically, the intersection over union (IoU) of each defect detection frame is calculated, that is, the ratio of the intersection and union of each defect detection frame and the defect detection frame with the highest confidence of the current detection frame. According to the IoU of each defect detection frame, the detection frame confidence of each defect detection frame is adjusted. When the IoU exceeds the preset ratio threshold, the detection frame confidence of the defect detection frame is reduced. If the IoU does not exceed the preset ratio threshold, the detection frame confidence of the defect detection frame is not adjusted. Specifically, the detection frame confidence adjustment formula is as follows:

[0227] Among them, N t It is expressed as a preset ratio threshold, IoU(M,b i ) is the intersection-over-union ratio of the defect detection frame, M is the defect detection frame with the highest confidence, b i Represents the current defect detection box, s i Represented as the current defect detection box b i The confidence of the detection box can be increased, so that overlapping and inconspicuous defects can be retained, reducing the probability of missed detection and false detection.

[0228] In one possible implementation, in the process of training the defect detection model based on the target classification loss, the coordinate difference of the defect detection frame can also be determined based on the coordinates of the defect detection frame in the sample detection frame information and the coordinates of the actual label detection frame marked in the training image; then, the regression loss of the defect detection frame is determined based on the coordinate difference; and then, the defect detection model is trained based on the regression loss and the target classification loss.

[0229] Therefore, by combining regression loss and target classification loss to train the defect detection model, using regression loss to perform fitting regression of the defect detection box coordinates, and using target classification loss to supervise and constrain the category of the defect detection box, it is possible to simultaneously guide the defect detection box regression and classification problems of the defect detection model, improve the performance of the defect detection model, and alleviate the impact of sample category imbalance in the defect detection model.

[0230] In one possible implementation, the sample detection frame information in the training image also includes the coordinates of the defect detection frame, so the difference in coordinates (x, y, w, h) between the defect detection frame predicted by the model and the actual label detection frame of the training image can be calculated, where the x element represents the horizontal coordinate of the center of the detection frame, the y element represents the vertical coordinate of the center of the detection frame, the w element represents the width of the detection frame, that is, the size along the horizontal direction of the image, and the h element represents the height of the detection frame, that is, the size along the vertical direction of the image.

[0231] Specifically, the defect detection frame and the actual label detection frame can be normalized to eliminate the effect of size and obtain more accurate coordinate differences. Then, a regression loss function, such as the Smooth L1 Loss function, can be used to calculate the regression loss of the defect detection frame based on the coordinate difference of the defect detection frame. The regression loss can then be used to perform fitting regression of the detection frame.

[0232] Specifically, the regression loss can be optimized using the gradient descent method. During the optimization process, the model parameters are continuously updated so that the defect detection frame predicted and output by the model gradually approaches the actual label detection frame. By minimizing the regression loss, the defect detection model can fit a more accurate defect detection frame.

[0233] In one possible implementation, referring to FIG18 , which is a schematic diagram of a defect detection process provided by another embodiment of the present application, the defect detection process of the target workpiece may include two branch processes, one of which may include the following steps:

[0234] 1801. Obtain a template image and an image to be measured of the target workpiece.

[0235] 1802. Perform image alignment between the image to be tested and the template image to generate a transformation matrix.

[0236] 1803. Rotate and translate the template image to align it with the image to be tested.

[0237] 1804. Compare the image to be tested with the template image to generate a differential image.

[0238] 1805. Perform morphological image processing on the difference image, such as dilation / erosion, and filter out noise areas (difference outliers caused by slight offset of the artifact).

[0239] 1806. Based on the logic algorithm, locate the defect area in the differential image and determine the first detection result.

[0240] Another branch of the process may include the following steps:

[0241] 1807. The image features of the differential image are concatenated with the image features of the image to be measured to form four-channel image feature data.

[0242] 1808. Use the defect detection model to perform defect detection and output defect detection box information.

[0243] 1809. Combine the first inspection result and the defect detection frame information for joint post-processing to determine a second inspection result for the target workpiece. In the step of combining the first inspection result and the defect detection frame information to determine the second inspection result for the target workpiece, the first inspection result and the defect detection frame information may be input into a joint post-processing model for quality inspection prediction, thereby outputting the second inspection result for the target workpiece.

[0244] In one possible implementation, in the process of inputting the image to be tested and the differential image into the defect detection model and extracting the target image features, the image to be tested and the differential image can be concatenated to obtain a concatenated image, and then the concatenated image is input into the defect detection model to extract the target image features of the concatenated image.

[0245] In one possible implementation, the image to be tested and the difference image are first scaled to ensure that the corresponding dimensions of the two images in the concatenation direction are consistent. The concatenation of the image to be tested and the difference image can be horizontal or vertical, that is, the two images are spliced ​​together along the horizontal direction of the images, or the two images are spliced ​​together along the vertical direction of the images.

[0246] Referring to Figure 19, which is a schematic diagram illustrating the defect detection frame information generation process provided by another embodiment of the present application, the image to be tested is a three-channel color RGB image, while the differential image can be a single-channel grayscale image. The image to be tested and the differential image are directly concatenated to form a concatenated image, where the concatenated image is a four-channel image.

[0247] Since there is no need to extract features from the image to be tested and the differential image separately, the loss caused by feature extraction is reduced. The concatenated images can retain the original feature information of the two images, which is helpful for model mining and learning, and can extract more accurate target image features. At the same time, the image concatenation method can be used to input the defect detection model, which can provide difference information of different images at the same position, helping to improve the accuracy of model detection.

[0248] Since the target image features are extracted based on the concatenated images, which are generated by splicing the image to be tested and the differential image, and the differential image carries the difference information between the target workpiece and the reference workpiece, the target image features can capture the key difference information, thereby enabling the target image features to assist the model in defect detection and improve the accuracy of the defect detection frame information.

[0249] The defect detection method provided by the embodiments of the present application is described in detail below.

[0250] 20 , which is a schematic diagram of an optional overall flow chart of a defect detection method provided in an embodiment of the present application, wherein the defect detection method includes but is not limited to the following steps 2001 to 2024:

[0251] Step 2001: Acquire an image of a target workpiece to be measured and a template image of the target workpiece.

[0252] Step 2002: Using a sliding window, determine a current window area in the image to be tested; and calculate a correlation coefficient between the image in the current window area and the image in the corresponding window area in the template image.

[0253] Step 2003: determining a target region in the image to be tested based on the correlation coefficients of the respective window regions; and determining a transformation matrix required to transform the target region into an aligned region in the template image.

[0254] Step 2004: Determine the inverse matrix of the transformation matrix, and use the inverse matrix as the transformation parameter between the image to be tested and the template image.

[0255] Step 2005: Align the target workpiece displayed in the template image with the target workpiece displayed in the image to be measured according to the transformation parameters to obtain an aligned image to be measured.

[0256] Step 2006: Compare the aligned image to be measured with the template image to obtain an initial differential image.

[0257] Step 2007: Perform morphological filtering on the initial differential image to obtain a differential image.

[0258] Step 2008: copy the data of each channel in the differential image until the number of channels in the copied differential image is equal to the number of channels in the image to be measured.

[0259] Step 2009: Input the image to be tested and the copied differential image into the defect detection model respectively.

[0260] In this step, the image to be tested and the copied differential image share the same feature extraction module in the defect detection model.

[0261] Step 2010: Perform multiple stages of convolution on the image to be tested in sequence.

[0262] In this step, each stage includes at least one convolution branch, and each convolution branch is used to perform multiple convolutions; when entering the next stage from the current stage, a new convolution branch is added, and the resolution of the new convolution branch is smaller than any existing convolution branch in the current stage; the input features of any convolution branch in the current stage are obtained by fusion of the features output by all convolution branches in the previous stage.

[0263] Step 2011: Obtain the initial convolution features output by each convolution branch in the last stage, fuse multiple initial convolution features to obtain fused features; for each of the multiple convolution branches in the last stage, perform convolution with the same resolution as the convolution branch on the fused features to obtain the second target convolution feature of the convolution branch.

[0264] In this step, during the last convolution, for any second convolution branch, the features of the current convolution output are upsampled or downsampled according to the resolution relationship between the current convolution branch and the remaining second convolution branches, and then added to the input of the last convolution of the remaining second convolution branches; then the initial convolution features of the last convolution output of each second convolution branch are obtained.

[0265] Step 2012: Fuse multiple second target convolution features to obtain the first target convolution feature.

[0266] Step 2013: down-sample the first target convolution feature to obtain a first image feature of the image to be tested.

[0267] Step 2014: Concatenate the feature value of each pixel in the first image feature with the feature value of the corresponding pixel in the second image feature to obtain a third image feature.

[0268] Step 2015: Perform dimensionality reduction on the third image feature to obtain a fourth image feature.

[0269] Step 2016: Activate the fourth image feature to obtain the target image feature.

[0270] Step 2017: performing defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested.

[0271] In this step, the defect detection model includes multiple cascaded network heads, a local area feature extractor connected to each network head, and a region proposal network. The target image features are input into the region proposal network for region extraction to obtain reference detection frame coordinates; for the first network head, the reference detection frame coordinates and the target image features are input into the local area feature extractor connected to the first network head for pooling, the obtained pooled features are input into the first network head for defect detection, and the defect detection frame coordinates are output; for each of the remaining network heads, the defect detection frame coordinates output by the previous network head and the target image features are input into the local area feature extractor connected to the network head for pooling, the obtained pooled features are input into the network head for defect detection, and the defect detection frame coordinates are output; wherein, the defect detection frame coordinates output by the last network head are determined as the defect detection frame information.

[0272] Step 2018: Locate the defect area in the differential image, and determine a first detection result of the target workpiece based on the defect area.

[0273] Step 2019: Determine whether the first detection result indicates that the target workpiece has defects. If so, execute step 2020; if not, execute step 2021.

[0274] Step 2020 : Determine that the second inspection result of the target workpiece is that the target workpiece has a defect, and execute step 2024 .

[0275] Step 2021: Obtain a preset confidence probability threshold and a preset area threshold corresponding to the target defect category.

[0276] Step 2022: Determine the defect area according to the defect detection frame coordinates, determine a first size relationship between the defect area and a preset area threshold, and a second size relationship between the detection frame confidence probability and the preset confidence probability threshold.

[0277] Step 2023: Determine a second detection result of the target workpiece according to the first size relationship and the second size relationship.

[0278] Step 2024: End the step process.

[0279] The defect detection method provided by another embodiment of the present application is described in detail below.

[0280] 21 , which is a schematic diagram of an optional overall flow chart of a defect detection method provided in an embodiment of the present application, wherein the defect detection method includes but is not limited to the following steps 2101 to 2120:

[0281] Step 2101: Acquire an image of a target workpiece to be measured and a template image of the target workpiece.

[0282] Step 2102: Using a sliding window, determine a current window area in the image to be tested; and calculate a correlation coefficient between the image in the current window area and the image in the corresponding window area in the template image.

[0283] Step 2103: Determine a target region in the image to be tested based on the correlation coefficients of the respective window regions; and determine a transformation matrix required to transform the target region into an aligned region in the template image.

[0284] Step 2104: Determine the inverse matrix of the transformation matrix, and use the inverse matrix as the transformation parameter between the image to be measured and the template image.

[0285] Step 2105: Align the target workpiece displayed in the template image with the target workpiece displayed in the image to be measured according to the transformation parameters to obtain an aligned image to be measured.

[0286] Step 2106: Compare the aligned image to be measured with the template image to obtain an initial differential image.

[0287] Step 2107: Perform morphological filtering on the initial differential image to obtain a differential image.

[0288] Step 2108: Concatenate the image to be tested and the differential image to obtain a concatenated image.

[0289] Step 2109: Input the concatenated images into the defect detection model and perform multiple stages of convolution on the concatenated images.

[0290] In this step, each stage includes at least one convolution branch, and each convolution branch is used to perform multiple convolutions; when entering the next stage from the current stage, a new convolution branch is added, and the resolution of the new convolution branch is smaller than any existing convolution branch in the current stage; the input features of any convolution branch in the current stage are obtained by fusion of the features output by all convolution branches in the previous stage.

[0291] Step 2110: Obtain the initial convolution features output by each convolution branch in the last stage, fuse the multiple initial convolution features to obtain the fused features; for each of the multiple convolution branches in the last stage, perform a convolution with the same resolution as the convolution branch on the fused features to obtain the second target convolution feature of the convolution branch.

[0292] In this step, during the last convolution, for any second convolution branch, the features of the current convolution output are upsampled or downsampled according to the resolution relationship between the current convolution branch and the remaining second convolution branches, and then added to the input of the last convolution of the remaining second convolution branches; then the initial convolution features of the last convolution output of each second convolution branch are obtained.

[0293] Step 2111: Fuse multiple second target convolution features to obtain the first target convolution feature.

[0294] Step 2112: Downsample the first target convolution feature to obtain the target image feature of the concatenated image.

[0295] Step 2113: Perform defect detection on the concatenated image based on the target image features to obtain defect detection frame information in the image to be tested.

[0296] In this step, the defect detection model includes multiple cascaded network heads, a local area feature extractor connected to each network head, and a region proposal network. The target image features are input into the region proposal network for region extraction to obtain reference detection frame coordinates; for the first network head, the reference detection frame coordinates and the target image features are input into the local area feature extractor connected to the first network head for pooling, the obtained pooled features are input into the first network head for defect detection, and the defect detection frame coordinates are output; for each of the remaining network heads, the defect detection frame coordinates output by the previous network head and the target image features are input into the local area feature extractor connected to the network head for pooling, the obtained pooled features are input into the network head for defect detection, and the defect detection frame coordinates are output; wherein, the defect detection frame coordinates output by the last network head are determined as the defect detection frame information.

[0297] Step 2114: Locate the defect area in the differential image, and determine a first detection result of the target workpiece based on the defect area.

[0298] Step 2115 : Determine whether the first detection result indicates that the target workpiece has a defect. If so, execute step 2116 ; if not, execute step 2117 .

[0299] Step 2116 : Determine that the second inspection result of the target workpiece is that the target workpiece has a defect, and execute step 2120 .

[0300] Step 2117: Obtain a preset confidence probability threshold and a preset area threshold corresponding to the target defect category.

[0301] Step 2118: Determine the defect area based on the defect detection frame coordinates, determine a first size relationship between the defect area and a preset area threshold, and a second size relationship between the detection frame confidence probability and the preset confidence probability threshold.

[0302] Step 2119: Determine a second detection result of the target workpiece based on the first size relationship and the second size relationship.

[0303] Step 2120: End the step process.

[0304] It will be appreciated that, although the various steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated in the present embodiment, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0305] 22 , which is a schematic diagram of an optional structure of a defect detection device 2200 provided in an embodiment of the present application, the defect detection device 2200 includes:

[0306] The first processing module 2201 is configured to obtain an image to be measured of a target workpiece and a template image of the target workpiece, and compare the image to be measured with the template image to obtain a differential image;

[0307] The second processing module 2202 is configured to input the image to be tested and the differential image into a defect detection model and perform the following processing: extracting target image features based on the image to be tested and the differential image; performing defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested;

[0308] A third processing module 2203 is configured to locate a defective area in the differential image and determine a first detection result of the target workpiece according to the defective area;

[0309] The fourth processing module 2204 is configured to determine a second detection result of the target workpiece according to the defect detection frame information and the first detection result.

[0310] Furthermore, the second processing module 2202 is configured to: extract a first image feature from the image to be tested; extract a second image feature from the differential image; and fuse the first image feature and the second image feature to obtain the target image feature.

[0311] In a possible implementation, the second processing module 2202 is configured to:

[0312] Copying the data of each channel in the differential image until the number of channels of the copied differential image is equal to the number of channels of the image to be measured;

[0313] The image to be tested and the copied differential image are respectively input into the defect detection model, wherein the image to be tested and the copied differential image share the same feature extraction module in the defect detection model.

[0314] In a possible implementation, the second processing module 2202 is configured to:

[0315] Multiple stages of convolution are sequentially performed on the image to be tested, wherein:

[0316] Each stage includes at least one convolution branch, and each convolution branch is used to perform multiple convolutions;

[0317] When entering the next stage from the current stage, a new convolution branch is added, wherein the resolution of the new convolution branch is smaller than any existing convolution branch in the current stage;

[0318] The input feature of any convolution branch in the current stage is obtained by fusion of the output features of all convolution branches in the previous stage;

[0319] The initial convolution features output by each convolution branch in the last stage are fused to obtain the first target convolution feature;

[0320] Downsample the first target convolution feature to obtain the first image feature.

[0321] In a possible implementation, the second processing module 2202 is configured to:

[0322] Fuse multiple initial convolution features to obtain fused features;

[0323] For each of the multiple convolution branches in the last stage, performing a convolution with the same resolution as the convolution branch on the fused features to obtain a second target convolution feature of the convolution branch;

[0324] The plurality of the second target convolution features are fused to obtain the first target convolution feature.

[0325] In a possible implementation, the second processing module 2202 is configured to:

[0326] cascading the feature value of each pixel in the first image feature with the feature value of the corresponding pixel in the second image feature to obtain a third image feature;

[0327] Performing dimensionality reduction on the third image feature to obtain a fourth image feature;

[0328] The fourth image feature is activated to obtain the target image feature.

[0329] In one possible implementation, the defect detection model includes multiple cascaded network heads, a local region feature extractor connected to each network head, and a region proposal network. The second processing module 2202 is used to:

[0330] Inputting the target image features into the region proposal network for region extraction to obtain reference detection frame coordinates;

[0331] For the first network header,

[0332] Inputting the reference detection frame coordinates and the target image features into a local area feature extractor connected to the first network head for pooling, inputting the obtained pooled features into the first network head for defect detection, and outputting the defect detection frame coordinates;

[0333] For each of the remaining network headers,

[0334] Inputting the defect detection frame coordinates and the target image features output by the previous network head into the local area feature extractor connected to the network head for pooling, inputting the obtained pooled features into the network head for defect detection, and outputting the defect detection frame coordinates;

[0335] The coordinates of the defect detection frame output by the last network head are determined as the defect detection frame information.

[0336] In a possible implementation, the defect detection frame information includes defect detection frame coordinates, detection frame confidence probability, and target defect category. The fourth processing module 2204 is configured to:

[0337] When the first detection result indicates that the target workpiece does not have defects, obtaining a preset confidence probability threshold and a preset area threshold corresponding to the target defect category;

[0338] Determining a defect area according to the defect detection frame coordinates, determining a first size relationship between the defect area and the preset area threshold, and a second size relationship between the detection frame confidence probability and the preset confidence probability threshold;

[0339] The second detection result is determined according to the first size relationship and the second size relationship.

[0340] In a possible implementation, the fourth processing module 2204 is further configured to:

[0341] When the first detection result indicates that the target workpiece has a defect, the second detection result is determined to be that the target workpiece has a defect.

[0342] In a possible implementation, the first processing module 220 is configured to:

[0343] Determining transformation parameters between the image to be measured and the template image;

[0344] Aligning the target workpiece displayed in the template image with the target workpiece displayed in the image to be measured according to the transformation parameters to obtain an aligned image to be measured;

[0345] Comparing the aligned image to be measured with the template image to obtain an initial differential image;

[0346] The initial differential image is filtered to obtain the differential image.

[0347] In a possible implementation, the first processing module 2201 is configured to:

[0348] Using a sliding window, determining a current window area in the image to be measured;

[0349] Calculating a correlation coefficient between the image in the current window area and the image in the corresponding window area in the template image;

[0350] determining a target area in the image to be measured according to the correlation coefficients of the respective window areas;

[0351] determining a transformation matrix required to transform the target region to an aligned region in the template image;

[0352] The inverse matrix of the transformation matrix is ​​determined as the transformation parameter.

[0353] Furthermore, the second processing module 2202 is configured to:

[0354] Concatenating the image to be measured and the differential image to obtain a concatenated image;

[0355] The concatenated images are input into the defect detection model to extract the target image features.

[0356] In a possible implementation, the defect detection apparatus further includes a first training module, where the first training module is configured to:

[0357] Acquire a training image of the target workpiece, wherein the training image is annotated with a corresponding defect category label;

[0358] Comparing the training image with the template image to obtain a sample difference image;

[0359] Inputting the training image and the sample difference image into the defect detection model to obtain sample detection frame information in the training image, wherein the sample detection frame information includes the sample defect category of the target workpiece;

[0360] Determining an initial classification loss based on the sample defect category and the defect category label;

[0361] Adjusting the initial classification loss according to a preset first adjustment parameter and a second adjustment parameter to obtain a target classification loss, wherein the first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the classification difficulty corresponding to the defect category label;

[0362] The defect detection model is trained according to the target classification loss.

[0363] The above-mentioned defect detection device 2200 and the defect detection method are based on the same inventive concept. The test image of the target workpiece is compared with the template image to obtain a differential image. The differential image can provide visual difference information of the target workpiece. The test image and the differential image are then input into the defect detection model, thereby introducing the difference information into the model, which helps to improve the accuracy of the model in feature extraction. The target image features are then extracted, and defect detection is performed on the test image based on the target image features to obtain defect detection frame information in the test image. By using the features corresponding to the differential image in the target image features to highlight the features related to the defect area in the test image, it helps to improve the model's detection ability for defect classification that has not been learned due to the lack of training samples, and improve the accuracy of the defect detection frame information. The defect area obtained by locating the differential image is used to determine the first detection result of the target workpiece, and the second detection result of the target workpiece is determined by combining the first detection result and the defect detection frame information. That is, based on the deep learning algorithm for defect detection and combining the features of multiple comparisons, the high visual visibility of the compared features can be used to visually assist defect detection, effectively improve the accuracy of defect detection, and reduce dependence on training samples.

[0364] The electronic device for performing the above-mentioned defect detection method provided in the embodiment of the present application may be a terminal. Referring to FIG23 , FIG23 is a partial structural block diagram of the terminal provided in the embodiment of the present application. The terminal includes components such as a camera assembly 2310, a memory 2320, an input unit 2330, a display unit 2340, a sensor 2350, an audio circuit 2360, a wireless fidelity (WiFi) module 2370, a processor 2380, and a power supply 2390. It will be understood by those skilled in the art that the terminal structure shown in FIG23 does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown, or some components may be combined, or the components may be arranged differently.

[0365] In this embodiment, the processor 2380 included in the terminal can execute the defect detection method of the previous embodiment.

[0366] The electronic device for performing the above-mentioned defect detection method provided in the embodiment of the present application can also be a server. Referring to FIG. 24 , FIG. 24 is a partial structural block diagram of a server provided in the embodiment of the present application. The server 2400 may vary significantly due to different configurations or performances, and may include one or more central processing units (CPUs) 2422 (e.g., one or more processors) and a memory 2432, and one or more storage media 2430 (e.g., one or more mass storage devices) storing application programs 2442 or data 2444. The memory 2432 and the storage medium 2430 may be either temporary storage or permanent storage. The program stored in the storage medium 2430 may include one or more modules (not shown), each of which may include a series of instruction operations on the server 2400. Furthermore, the CPU 2422 may be configured to communicate with the storage medium 2430 to execute the series of instruction operations in the storage medium 2430 on the server 2400.

[0367] The server 2400 may also include one or more power supplies 2424, one or more wired or wireless network interfaces 2450, one or more input and output interfaces 2458, and / or one or more operating systems 2441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0368] The processor in the server 2400 may be configured to execute the defect detection method.

[0369] An embodiment of the present application further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the defect detection methods of the aforementioned embodiments.

[0370] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to implement the above-mentioned defect detection method.

[0371] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the numbers used in this manner are interchangeable where appropriate to describe embodiments of the present application, for example, capable of being implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0372] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0373] It should be understood that in the description of the embodiments of the present application, multiple (or multiple items) means more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0374] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0375] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0376] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0377] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0378] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0379] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A defect detection method, performed by an electronic device, comprising: Acquire a to-be-tested image of a target workpiece and a template image of the target workpiece, and compare the to-be-tested image with the template image to obtain a differential image; The image to be tested and the differential image are input into the defect detection model, and the following processing is performed: Extracting target image features based on the image to be tested and the differential image; Perform defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested; Locating a defective area in the differential image, and determining a first detection result of the target workpiece according to the defective area; and, A second detection result of the target workpiece is determined according to the defect detection frame information and the first detection result.

2. The defect detection method according to claim 1, wherein: The step of extracting target image features based on the image to be tested and the differential image includes: Extracting a first image feature from the image to be tested; extracting a second image feature from the difference image; The first image feature and the second image feature are fused to obtain the target image feature.

3. The defect detection method according to claim 2, wherein: The step of inputting the image to be tested and the differential image into a defect detection model comprises: Copying the data of each channel in the differential image until the number of channels of the copied differential image is equal to the number of channels of the image to be tested; The image to be tested and the copied differential image are respectively input into the defect detection model, wherein the image to be tested and the copied differential image share the same feature extraction module in the defect detection model.

4. The defect detection method according to claim 2, wherein: The step of extracting a first image feature from the image to be tested includes: Multiple stages of convolution are sequentially performed on the image to be tested, wherein: Each stage includes at least one convolution branch, and each convolution branch is used to perform multiple convolutions; When entering the next stage from the current stage, a new convolution branch is added, wherein the resolution of the new convolution branch is smaller than any existing convolution branch in the current stage; The input feature of any convolution branch in the current stage is obtained by fusion of the output features of all convolution branches in the previous stage; The initial convolution features output by each convolution branch in the last stage are fused to obtain the first target convolution feature; The first target convolution feature is downsampled to obtain the first image feature.

5. The defect detection method according to claim 4, wherein: The initial convolution features output by each convolution branch in the last stage are fused to obtain the first target convolution features, including: Fuse multiple initial convolution features to obtain fused features; For each of the multiple convolution branches in the last stage, performing a convolution with the same resolution as that of the convolution branch on the fused features to obtain a second target convolution feature of the convolution branch; A plurality of the second target convolution features are fused to obtain the first target convolution feature.

6. The defect detection method according to claim 2, wherein: The fusing the first image feature and the second image feature to obtain the target image feature includes: Cascading the feature value of each pixel in the first image feature with the feature value of the corresponding pixel in the second image feature to obtain a third image feature; Performing dimensionality reduction on the third image feature to obtain a fourth image feature; The fourth image feature is activated to obtain the target image feature.

7. The defect detection method according to any one of claims 1 to 6, wherein: The defect detection model includes a plurality of cascaded network heads, a local area feature extractor connected to each network head, and a region proposal network. The defect detection is performed on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested, including: Inputting the target image features into the region proposal network for region extraction to obtain reference detection frame coordinates; For the first network header, Inputting the reference detection frame coordinates and the target image features into a local area feature extractor connected to the first network head for pooling, inputting the obtained pooled features into the first network head for defect detection, and outputting the defect detection frame coordinates; For each of the remaining network headers, Inputting the defect detection frame coordinates and the target image features output by the previous network head into a local area feature extractor connected to the network head for pooling, inputting the obtained pooled features into the network head for defect detection, and outputting the defect detection frame coordinates; Among them, the defect detection frame coordinates output by the last network head are determined as the defect detection frame information.

8. The defect detection method according to any one of claims 1 to 7, wherein: The defect detection frame information includes defect detection frame coordinates, detection frame confidence probability, and target defect category, and determining a second detection result of the target workpiece according to the defect detection frame information and the first detection result includes: When the first detection result indicates that the target workpiece has no defects, obtaining a preset confidence probability threshold and a preset area threshold corresponding to the target defect category; Determine a defect area according to the defect detection frame coordinates, determine a first size relationship between the defect area and the preset area threshold, and a second size relationship between the detection frame confidence probability and the preset confidence probability threshold; The second detection result is determined according to the first size relationship and the second size relationship.

9. The defect detection method according to claim 8, further comprising: When the first detection result indicates that the target workpiece has a defect, the second detection result is determined as that the target workpiece has a defect.

10. The defect detection method according to any one of claims 1 to 9, wherein: The step of comparing the image to be tested with the template image to obtain a differential image includes: Determining transformation parameters between the image to be tested and the template image; According to the transformation parameters, aligning the target workpiece displayed in the template image with the target workpiece displayed in the image to be measured to obtain an aligned image to be measured; Comparing the aligned image to be tested with the template image to obtain an initial differential image; The initial differential image is filtered to obtain the differential image.

11. The defect detection method according to claim 10, wherein: The determining of the transformation parameters between the image to be tested and the template image comprises: Using a sliding window, determining a current window area in the image to be tested; Calculating a correlation coefficient between the image in the current window area and the image in the corresponding window area in the template image; Determining a target area in the image to be tested according to correlation coefficients of the respective window areas; determining a transformation matrix required to transform the target region to an aligned region in the template image; The inverse matrix of the transformation matrix is ​​determined as the transformation parameter.

12. The defect detection method according to claim 1, wherein: The step of extracting target image features based on the image to be tested and the differential image includes: Concatenating the image to be tested and the differential image to obtain a concatenated image; The concatenated images are input into the defect detection model to extract the target image features.

13. The defect detection method according to any one of claims 1 to 12, wherein: The defect detection model is trained by the following steps: Acquire a training image of the target workpiece, wherein the training image is annotated with a corresponding defect category label; Comparing the training image with the template image to obtain a sample difference image; Inputting the training image and the sample difference image into the defect detection model to obtain sample detection frame information in the training image, wherein the sample detection frame information includes a sample defect category of the target workpiece; Determining an initial classification loss according to the sample defect category and the defect category label; According to the preset first adjustment parameter and second adjustment parameter, the initial classification loss is adjusted to obtain the target classification loss, wherein the first adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label, and the second adjustment parameter is related to the number of positive and negative samples corresponding to the defect category label. The labels are related to the difficulty of classification; The defect detection model is trained according to the target classification loss.

14. A defect detection device comprising: A first processing module is used to obtain a to-be-tested image of a target workpiece and a template image of the target workpiece, and compare the to-be-tested image with the template image to obtain a differential image; A second processing module is used to input the image to be tested and the differential image into a defect detection model, and perform the following processing: extracting target image features based on the image to be tested and the differential image; Perform defect detection on the image to be tested based on the target image features to obtain defect detection frame information in the image to be tested; A third processing module, configured to locate a defect area in the differential image and determine a first detection result of the target workpiece according to the defect area; The fourth processing module is used to determine a second detection result of the target workpiece according to the defect detection frame information and the first detection result.

15. The defect detection device according to claim 14, wherein: The second processing module is used to extract a first image feature from the image to be tested; extracting a second image feature from the difference image; The first image feature and the second image feature are fused to obtain the target image feature.

16. The defect detection device according to claim 14 or 15, wherein: The defect detection model includes multiple cascaded network heads, a local area feature extractor connected to each network head, and a region proposal network. The second processing module is used to input the target image features into the region proposal network for region extraction to obtain reference detection frame coordinates; for the first network head, the reference detection frame coordinates and the target image features are input into the local area feature extractor connected to the first network head for pooling, the obtained pooled features are input into the first network head for defect detection, and the defect detection frame coordinates are output; for each of the remaining network heads, the defect detection frame coordinates output by the previous network head and the target image features are input into the local area feature extractor connected to the network head for pooling, the obtained pooled features are input into the network head for defect detection, and the defect detection frame coordinates are output; wherein the defect detection frame coordinates output by the last network head are determined as the defect detection frame information.

17. The defect detection device according to any one of claims 14 to 16, wherein: The defect detection frame information includes defect detection frame coordinates, detection frame confidence probability and target defect category. The fourth processing module is used to obtain a preset confidence probability threshold and a preset area threshold corresponding to the target defect category when the first detection result indicates that there is no defect in the target workpiece; determine the defect area according to the defect detection frame coordinates, determine a first size relationship between the defect area and the preset area threshold, and determine a second size relationship between the detection frame confidence probability and the preset confidence probability threshold; determine the second detection result according to the first size relationship and the second size relationship.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the defect detection method according to any one of claims 1 to 13 when executing the computer program.

19. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the defect detection method according to any one of claims 1 to 13.

20. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the defect detection method according to any one of claims 1 to 13 is implemented.

Images