An optical detection method and system based on double-path fusion and feedback self-learning

Abstract

The present application relates to the technical field of optical automatic detection, and particularly relates to an optical detection method and system based on double-path fusion and feedback self-learning, the detection method comprising: training a lightweight real-time backbone reasoning model, and constructing a normal image block memory bank; collecting a to-be-detected image and inputting the to-be-detected image into the lightweight real-time backbone reasoning model, and outputting a first abnormal heat map and a first abnormal score; adopting a risk gate scheduling method to determine the risk level of the to-be-detected image, and outputting the to-be-detected image as a normal image or a to-be-precisely-detected image; based on the normal image block memory bank, outputting a second abnormal heat map and a second abnormal score corresponding to the to-be-precisely-detected image; combining the first abnormal score and the second abnormal score to output an abnormal defect determination result; manually submitting a missed detection image and a false alarm image, and updating the lightweight real-time backbone reasoning model and the normal image block memory bank; and the detection system is applied to the above detection method, and can balance the detection speed and the detection precision under the condition of few samples.

Description

Technical Field

[0001] This invention relates to the field of optical automatic inspection technology, specifically to an optical inspection method and system based on dual-path fusion and feedback self-learning. Background Technology

[0002] In production scenarios such as electronic manufacturing, precision assembly, screen modules, battery modules, mid-frames, and connectors, optical automatic inspection methods are often used to replace or assist manual quality inspection in order to detect abnormal defects such as surface defects, texture anomalies, structural anomalies, and assembly anomalies during production. Existing optical automatic inspection methods mainly include the following categories:

[0003] One approach is rule-based and traditional machine vision methods, such as threshold segmentation, edge detection, template matching, and morphological analysis. These methods can effectively detect known fixed defects, but when encountering non-fixed defects, it is often necessary to manually redesign the rules and redevelop the detection methods. They are also less adaptable to new defects, new textures, new workstations, and changes in lighting.

[0004] Second, there are deep learning-based supervised detection or segmentation methods. These methods rely on a large number of labeled defect samples, but real defect samples in industrial scenarios are scarce, and it is difficult to collect enough data in time when new defects appear, making it difficult to apply them directly to real industrial scenarios.

[0005] Thirdly, there are anomaly detection methods based on unsupervised or one-class learning. For example, normal distribution is learned through normal samples, and then areas that deviate from the normal distribution are identified as abnormal defect areas. These methods do not rely on the collection and labeling of defect samples, but at the same time, they also result in insufficient utilization of available defect samples, lack of feedback loop, and difficulty in autonomous learning and updating.

[0006] In addition, some optical automatic detection methods have introduced human feedback systems to collect manually labeled detection results feedback data. However, the feedback data is only used as offline retraining data and lacks an online closed-loop mechanism that links with the optical automatic detection method in real time.

[0007] Therefore, it is necessary to provide an optical inspection method and system based on dual-path fusion and feedback self-learning, which can make full use of the limited number of defect samples in real industrial scenarios, can be autonomously updated based on closed-loop feedback, and has a stronger adaptability to new defects, new workstations, and new textures, and can quickly keep up with the quality inspection needs of new defects and new products. Summary of the Invention

[0008] To address the aforementioned problems, this invention provides an optical detection method and system based on dual-path fusion and feedback self-learning, thereby resolving the technical issues raised in the background section.

[0009] The technical solution adopted by this invention to solve its technical problem is as follows:

[0010] An optical detection method based on dual-path fusion and feedback self-learning includes the following steps:

[0011] S1: Collect normal sample images and defective sample images. Based on the defective sample images, train a lightweight real-time backbone inference model. Based on the normal sample images, construct a normal image block memory.

[0012] S2: Collect images or video frames as images to be detected, preprocess them, input them into a lightweight real-time backbone inference model, and output the corresponding first anomaly heatmap and first anomaly score;

[0013] S3: Based on the first anomaly score, a risk-gated scheduling method is used to determine whether the image to be detected is a low-risk image, a medium-risk image, or a high-risk image; for low-risk images, they are output as normal images; for medium-risk images, suspicious sub-regions are extracted as images to be further examined; for high-risk images, the entire image is retained as images to be further examined.

[0014] S4: Based on the normal image block memory, output the second anomaly heatmap and the second anomaly score corresponding to the image to be examined;

[0015] S5: Introduce a video frame stability detection method, combine the first anomaly score and the second anomaly score, output a normal image or an abnormal defect image, fuse the first anomaly heatmap and the second anomaly heatmap, and output an interpretable result;

[0016] S6: Based on interpretable results, manually submit missed images found in normal images and false alarm images found in abnormal defect images. Missed images are added to defect samples and the lightweight real-time backbone inference model is updated. False alarm images are added to normal samples and the normal image block memory is updated.

[0017] Furthermore, the specific method for training the lightweight real-time backbone inference model in S1 is as follows:

[0018] Before a new product, new workstation, or new inspection task is launched, a small number of defect sample images are collected.

[0019] Generate the corresponding defect mask for the defect sample image, and add the defect sample image and its corresponding defect mask to the defect sample set;

[0020] A lightweight real-time backbone inference model is constructed, including a convolutional encoder, a feature fusion layer, and a prediction head. The defect sample image is used as the input image during training, and the corresponding defect mask of the defect sample image is used as the constraint. The loss function is constructed by pixel classification loss and region overlap loss, and the lightweight real-time backbone inference model is trained.

[0021] Furthermore, the specific method for constructing the normal image block memory in S1 is as follows:

[0022] Before a new product, new workstation, or new testing task is launched, a small number of normal sample images are collected and added to the normal sample set.

[0023] For each normal sample image: a pre-defined convolutional neural network is used to extract its multi-layer feature map, and the multi-layer feature map is divided into multiple block features according to its spatial location; all block features are normalized and compressed to obtain normal block features, and the normal block features are saved to the normal image block memory.

[0024] Furthermore, the specific method for outputting the first anomaly heatmap and the first anomaly score in S2 is as follows: input the input image to be detected into the lightweight real-time backbone inference model to generate an anomaly response map corresponding to the input image to be detected, and use it as the first anomaly heatmap. The maximum value of the abnormal response in the first abnormal heatmap is selected as the first abnormal score. , that is,

[0025] Where p represents the pixel position in the first anomaly heatmap. This represents the entire first anomaly heatmap region.

[0026] Furthermore, the specific method for determining low-risk, medium-risk, or high-risk images in S3 is as follows:

[0027] Obtain the first anomaly score of the image to be detected. Set a low-risk threshold and high risk threshold , ;

[0028] like If so, the corresponding image to be detected is determined to be a low-risk image;

[0029] like If so, the corresponding image to be detected is determined to be a medium-risk image;

[0030] like If the image is detected, it is determined to be a high-risk image.

[0031] Furthermore, the specific method of S4 is as follows:

[0032] Multiple feature blocks to be refined are extracted from each image to be refined.

[0033] Calculate the Euclidean distance between each feature of the block to be refined and each normal block feature in the normal image block memory. The minimum Euclidean distance is used to obtain the memory anomaly score for each feature of the block to be refined, i.e.:

[0034]

[0035] in, , Let be the memory anomaly score and feature value of the i-th feature block to be refined, respectively. Let i be the feature value of the j-th normal block feature, i = 1, 2, 3, ..., j = 1, 2, 3, ...;

[0036] The anomaly scores of each feature block to be refined are used to generate a second anomaly heatmap based on their spatial location in the image to be refined. , ;

[0037] Extract the second anomaly score from the second anomaly heatmap. ,

[0038]

[0039] in, This refers to the pixel location in the second anomaly heatmap. This represents the entire area of ​​the second anomaly heatmap.

[0040] Furthermore, the video frame stability detection method in S5 is specifically as follows:

[0041] For the image to be detected originating from the t-th interval frame, the first and second anomaly heatmaps are fused to generate a fused anomaly heatmap. The fused anomaly heatmap is then processed using an exponential moving average to obtain a smoothed anomaly heatmap, i.e.

[0042]

[0043] Where t is a positive integer, This is a smoothed anomaly heatmap of the image to be detected. This is a smoothed anomaly heatmap of the previous frame in the image to be detected. This is the smoothing coefficient, and its value range is (0,1).

[0044] Obtain the anomaly detection results of the image to be detected and its n adjacent interval frames, perform short-window voting, and obtain the smoothed voting score of the image to be detected. ,

[0045]

[0046] Where n is a positive integer ranging from [3, 5]. The result of anomaly detection is given for k adjacent frames of the image to be detected.

[0047] The maximum response value in the smooth anomaly heatmap is selected as the smooth anomaly score. The video frame stability score is obtained by combining the smoothed voting score. , .

[0048] Furthermore, the specific method for outputting normal images or abnormal defect images and interpretable results in S5 is as follows:

[0049] Introducing process rule scores Semantic prior scores The final anomaly score is calculated by combining the first anomaly score, the second anomaly score, and the video frame stability score. ,

[0050]

[0051] in, , , , , These are the first weight, the second weight, the process rule weight, the video frame stabilization weight, and the semantic prior weight, respectively. Their values ​​are all in the range of [0,1], and the sum of their weights is 1. The value of each weight is determined by the detection mode and requirements.

[0052] Set the final judgment threshold ,when When the image to be detected is determined to be a normal image, it is output. When the image to be detected is determined to be an abnormal defect image and output, the final determination threshold value range is (0,1);

[0053] By integrating the first anomaly heatmap, the second anomaly heatmap, the process rule diagram, and the smoothed anomaly heatmap, the final anomaly heatmap is obtained. ,

[0054]

[0055] in, , , , The heatmap fusion weights are the weight coefficients of the first abnormal heatmap, the second abnormal heatmap, the process rule graph, and the smooth abnormal heatmap, respectively. Their values ​​are all in the range of [0,1], and the sum of each weight coefficient is 1. The process rule graph is obtained by constraining the image to be detected according to the process rules.

[0056] Defect boxes are obtained from the final anomaly heatmap, and interpretable results are output, including the original image of the image to be detected, the final anomaly heatmap, the overlay of the original image and the final anomaly heatmap, the defect boxes, the final anomaly score, the low-risk threshold, the high-risk threshold, the final judgment threshold, the detection inference time, and the structured JSON of the above results.

[0057] An optical detection system based on dual-path fusion and feedback self-learning, applied to the aforementioned optical detection method based on dual-path fusion and feedback self-learning, includes:

[0058] Lightweight inference module: used to train a lightweight real-time backbone inference model using defect sample images, receive preprocessed images to be detected, and output the first anomaly heatmap and the first anomaly score;

[0059] Risk gating scheduling module: used to receive the first anomaly score, determine whether the image to be detected is a low-risk image, a medium-risk image or a high-risk image, and output normal images and images to be further inspected;

[0060] The detailed inspection reasoning module is used to receive the image to be inspected, utilize the normal image block memory library, and output the second anomaly heatmap and the second anomaly score.

[0061] Dual-path fusion module: used to receive the first abnormal heatmap, the first abnormal score, the second abnormal heatmap and the second abnormal score, and output a normal image or an abnormal defect image, and output interpretable results;

[0062] Optical automated inspection platform: used to receive and display normal images, abnormal defect images, and interpretable results; to provide feedback on missed and false alarm images found by the operator; and to update the lightweight real-time backbone inference model, normal image block memory, and various thresholds.

[0063] Compared with the prior art, the beneficial effects of the present invention are:

[0064] 1. This invention provides an optical detection method based on dual-path fusion and feedback self-learning. It fuses a fast inference path using a lightweight real-time backbone inference model with a fine-check inference path using a normal image block memory. First, the lightweight real-time backbone inference model performs fast detection on the image to be detected under limited sample conditions. Then, a risk-gated scheduling method is used to identify medium-risk or high-risk images. Fine-check inference is then performed using the normal image block memory. The results of the two detection and inference processes are fused to generate the final abnormal defect judgment result, which ensures both detection efficiency and enhances the accuracy of the detection results.

[0065] 2. The present invention provides an optical detection method based on dual-path fusion and feedback self-learning. In the process of generating the final abnormal defect judgment result, a video frame stabilization method is introduced, which not only maintains the sensitivity of single-frame response, but also reduces the detection error caused by jitter, reflection and temporary occlusion of single-frame image to be detected.

[0066] 3. The present invention provides an optical inspection system based on dual-path fusion and feedback self-learning. Operators can use the optical automatic inspection platform to provide timely feedback on missed and false images of the system. The system updates itself based on the feedback, realizing adaptive optimization of the inspection system. This improves the system's adaptability to new defects, new workstations, and new textures, making it easier to apply in actual industrial scenarios. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the process of an optical detection method based on dual-path fusion and feedback self-learning as described in this embodiment;

[0069] Figure 2 This is a flowchart illustrating S3 as described in this embodiment;

[0070] Figure 3 This is a flowchart illustrating S6 as described in this embodiment;

[0071] Figure 4 This is a schematic diagram of an optical detection system based on dual-path fusion and feedback self-learning as described in this embodiment;

[0072] Figure 5 This is a schematic diagram of the interface of the optical automatic inspection platform described in this embodiment;

[0073] Figure 6 This is a schematic diagram of a key area of ​​a certain industrial product described in this embodiment. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] This invention provides an optical detection method based on dual-path fusion and feedback self-learning, such as... Figure 1 As shown, the method includes the following steps:

[0076] S1: Collect normal sample images and defective sample images. Based on the defective sample images, train a lightweight real-time backbone inference model. Based on the normal sample images, construct a normal image block memory.

[0077] Furthermore, the specific method for training the lightweight real-time backbone inference model in S1 is as follows:

[0078] Before a new product, new workstation, or new testing task is launched, a small number of defect sample images are collected. In this embodiment, the number of defect sample images collected is 30.

[0079] Generate the corresponding defect mask for the defect sample image, and add the defect sample image and its corresponding defect mask to the defect sample set. The defect mask of the defect sample image can be obtained by manual and precise annotation or by weak annotation using the manual bounding box method. When the defect location cannot be determined manually, an approximate defect mask can be generated by a weakly supervised algorithm. The specific mask generation is an existing technology.

[0080] A lightweight real-time backbone inference model is constructed, including a convolutional encoder, a feature fusion layer, and a prediction head. The convolutional encoder is used to generate multi-scale features of the input image, and the prediction head outputs a pixel-level anomaly response map corresponding to the input image. The defect sample image is used as the input image during training, and the corresponding defect mask of the defect sample image is used as a constraint. A loss function is constructed using pixel classification loss and region overlap loss to train the lightweight real-time backbone inference model.

[0081] Furthermore, the specific method for constructing the normal image block memory in S1 is as follows:

[0082] Before a new product, new workstation, or new testing task is launched, a small number of normal sample images are collected and added to the normal sample set. In this embodiment, the number of normal sample images collected is 100.

[0083] For each normal sample image: a pre-defined convolutional neural network is used to extract its multi-layer feature maps, and the multi-layer feature maps are divided according to their spatial location. Each block feature, , These represent the height and width dimensions of the multi-layer feature map, respectively. All block features are normalized to obtain normalized block features. The number of normalized block features is compressed using coreset sampling or random sampling to obtain normal block features. The normal block features are then saved to the normal image block memory.

[0084] In this embodiment, the preset convolutional neural network is a pre-trained ResNet18 convolutional neural network. The specific method for extracting multi-layer feature maps is as follows: normal sample images are scaled and normalized and then input into the convolutional up-down network to obtain intermediate feature maps output by the second residual stage and the third residual stage respectively; the second residual stage is used for local details such as surface texture and edges, and the third residual stage is used to represent structural information under a larger receptive field; the two intermediate feature maps are unified to the same spatial size through bilinear interpolation, and then spliced ​​along the channel direction to obtain the fused multi-layer feature map.

[0085] S2: Collect images or video frames as images to be detected, preprocess them, input them into a lightweight real-time backbone inference model, and output the corresponding first anomaly heatmap and first anomaly score;

[0086] Furthermore, the specific method for acquiring and preprocessing the image to be detected in S2 is as follows:

[0087] An industrial camera is used to collect images or videos of the inspection area. The resolution of the industrial camera is preferably 2500×2500. For the collected video, frames are extracted at fixed intervals to obtain video frames. The images or video frames of the inspection area are used as the images to be inspected. In this embodiment, the number of interval frames is 5 frames.

[0088] The image to be detected is converted into an RGB format image or a grayscale format image, and normalization processing is performed to obtain a normalized image to be detected.

[0089] The normalized image to be detected is scaled to a preset input size to obtain the input image to be detected. The preset input size is, for example, 256×256, 512×512, or 1024×1024.

[0090] Furthermore, the specific method for outputting the first anomaly heatmap and the first anomaly score in S2 is as follows: the input image to be detected is input into the lightweight real-time backbone inference model, and the lightweight real-time backbone inference model uses its convolutional encoder, feature fusion layer, and prediction head to generate an anomaly response map corresponding to the input image to be detected, which is used as the first anomaly heatmap. Extract the maximum value of the abnormal response value, the average value of the high abnormal response values, or the weighted average value of the abnormal response values ​​from the first abnormal heatmap as the first abnormal score. In this embodiment, the maximum value of the abnormal response value in the first abnormal heat map is preferably used as the first abnormal score. , that is,

[0091] Where p represents the pixel position in the first anomaly heatmap. This represents the entire first anomaly heatmap region. The value range is [0,1].

[0092] S3: Based on the first anomaly score, a risk-gated scheduling method is used to determine whether the image to be detected is a low-risk image, a medium-risk image, or a high-risk image; for low-risk images, they are output as normal images; for medium-risk images, suspicious sub-regions are extracted as images to be further examined; for high-risk images, the entire image is retained as images to be further examined.

[0093] Furthermore, such as Figure 2 As shown, the specific method of S3 is as follows:

[0094] Obtain the first anomaly score of the image to be detected. Set a low-risk threshold and high risk threshold , , Preferred ;

[0095] like If the image to be detected is determined to be a low-risk image, it will be output directly as a normal image.

[0096] like If the image to be detected is determined to be a medium-risk image, a suspicious sub-region is extracted and enlarged, and used as the image to be further examined. Specifically, in one embodiment, the method for extracting the suspicious sub-region is as follows: setting a threshold value. The first anomaly heatmap corresponding to the image to be detected is normalized, and the pixel regions with response values ​​greater than or equal to the threshold of the value frame are divided into binary candidate masks. Four-neighbor connected component analysis is performed on the binary candidate masks to obtain candidate connected components. The bounding rectangle of the candidate connected components is expanded according to a preset magnification factor r to obtain suspicious sub-regions. In this embodiment... r=1.5;

[0097] In another embodiment, the suspicious sub-region is generated from a key region preset by the production process, for example... Figure 6 As shown, for a certain industrial product with a complex textured background, the key area is a rectangular area located from the lower left to the middle. By enlarging the suspicious sub-region, it is beneficial to improve the accuracy of subsequent fine inspection. In this embodiment, the suspicious sub-region is enlarged to the resolution of the image to be detected before scaling.

[0098] like If the image is deemed high-risk, its entirety is retained as the image to be further examined.

[0099] Furthermore, through the aforementioned risk gating scheduling method, for some images to be detected that have abnormal defects, only suspicious sub-regions are extracted for detailed inspection, so that most normal images can be detected at high speed, while suspicious abnormal images can be located more precisely, reducing the time consumed by subsequent full-image detailed inspection, saving computing resources, and improving detection efficiency.

[0100] S4: Based on the normal image block memory, output the second anomaly heatmap and the second anomaly score corresponding to the image to be examined;

[0101] Furthermore, the specific method of S4 is as follows:

[0102] For each image to be finely examined: extract its multi-layer feature map, divide the multi-layer feature map into multiple block features, and after normalization and compressed sampling, obtain multiple block features to be finely examined. The specific method for obtaining the block features to be finely examined from the image to be finely examined is the same as the method for obtaining normal block features from normal sample images, and will not be described again.

[0103] Calculate the distance between each feature to be refined and its neighboring normal block features in the normal image block memory to obtain the memory anomalous score of each feature to be refined, i.e.:

[0104]

[0105] in, , Let be the memory anomaly score and feature value of the i-th feature block to be refined, respectively. Let i be the feature value of the j-th normal block feature, i = 1, 2, 3, ..., j = 1, 2, 3, ...;

[0106] The anomaly scores of each feature block to be refined are used to generate a second anomaly heatmap based on their spatial location in the image to be refined. , ;

[0107] Extract the second anomaly score from the second anomaly heatmap. ,

[0108]

[0109] in, This refers to the pixel location in the second anomaly heatmap. This represents the entire area of ​​the second anomaly heatmap.

[0110] S5: Introduce a video frame stability detection method, combine the first anomaly score and the second anomaly score, output a normal image or an abnormal defect image, fuse the first anomaly heatmap and the second anomaly heatmap, and output an interpretable result;

[0111] Furthermore, the video frame stability detection method in S5 is specifically as follows:

[0112] For the image to be detected originating from the t-th interval frame, the first and second anomaly heatmaps are fused to generate a fused anomaly heatmap. ,

[0113]

[0114] in, To merge pixel locations in anomaly heatmaps, The normalized response of the first anomaly heatmap. The normalized response of the second anomaly heatmap is... The normalized response of the process rule map is obtained by the following method: converting the image to be detected into a grayscale image, using Gaussian filtering to obtain a smooth background image, calculating the local difference between the original grayscale image and the smooth background image to obtain a local contrast response map, and simultaneously detecting overexposure or underexposure responses in near-black or white saturated regions to obtain a saturation response map. The local contrast response map and the saturation response map are then weighted and fused to obtain the process rule map. ;

[0115] , , The weights assigned to the first abnormal heat map, the second abnormal heat map, and the process rule branch are all within the range of [0,1]. , These are the enable indicator variables for the normal image block memory and the process rule branch, respectively. When the normal image block memory is invoked... Set the value to 1, otherwise set it to 0. When the process rule branch is invoked... Select 1 if the value is 1, otherwise select 0.

[0116] By fusing these anomaly heatmaps with an exponential moving average, a smoothed anomaly heatmap is obtained, i.e.,

[0117]

[0118] Where t is a positive integer, This is a smoothed anomaly heatmap of the image to be detected. This is a smoothed anomaly heatmap of the previous frame in the image to be detected. The smoothing coefficient has a value range of (0,1). In this embodiment, =0.6;

[0119] Obtain the anomaly detection results of the image to be detected and its n adjacent interval frames, perform short-window voting, and obtain the smoothed voting score of the image to be detected. ,

[0120]

[0121] Where n is a positive integer ranging from [3, 5]. The anomaly detection results are obtained from k adjacent frames of the image to be detected. , Let be the set of all pixel locations in the fused heatmap of the tk-th interval frame. This is the threshold for binary judgment. , This is an indicator function; the value is 1 if the condition within the indicator function is true, and 0 otherwise.

[0122] Based on the smoothing anomaly heatmap of the image to be detected, its smoothing anomaly score is obtained, and combined with the smoothing voting score, the video frame stability score is obtained. , By combining smoothed anomaly scores and smoothed voting scores, the sensitivity of single-frame response is maintained while reducing detection errors caused by jitter, reflection, and temporary occlusion in a single frame of the image to be detected.

[0123] Furthermore, the specific method for outputting normal images or abnormal defect images in S5 is as follows:

[0124] Introducing process rule scores The method for determining the process rule score is as follows: select high-response pixel regions from the process rule map, calculate their average response value, and use it as the process rule score;

[0125] Introducing semantic prior scores The method for determining the semantic prior score is as follows: collect the semantic information generated during the reasoning process of the lightweight real-time backbone inference model, the risk gating scheduling process, and the reasoning process based on the normal image block memory, perform semantic prior on the abnormal risk of the image to be detected, and generate a semantic prior score with a value range of (0,1).

[0126] The final anomaly score is calculated by combining the first anomaly score, the second anomaly score, the video frame stability score, the process rule score, and the semantic prior score. ,

[0127]

[0128] in, , , , , These are the first weight, the second weight, the process rule weight, the video frame stabilization weight, and the semantic prior weight, each with a value range of [0,1]. The sum of their weights is 1. The value of each weight is determined by the detection mode and requirements. Specifically:

[0129] In high-speed online detection mode, the first weight is increased, and the low-risk and high-risk thresholds are decreased to reduce the frequency of calls to the normal image block memory and ensure detection speed. In high-precision composite mode, the second weight and process rule weight are increased to obtain more reliable defect localization results. When processing video streams, the video frame stability weight is increased to enhance the stability of continuous detection results. In the rapid deployment phase of new products or new workstations, the second weight and semantic prior weight are increased to enhance few-sample generalization ability. Preferably, ;

[0130] Set the final judgment threshold ,when When the image to be detected is determined to be a normal image, it is output. When the image to be detected is determined to be an abnormal defect image and output, for the image to be detected that is determined to be a medium-risk image, the output abnormal defect image is a suspicious sub-region. The final determination threshold range is (0,1). By fusing a lightweight real-time backbone inference model and a normal image block memory, defect detection is performed on the image to be detected, which can enhance the few-sample generalization ability while ensuring the detection speed.

[0131] Furthermore, the specific method for outputting interpretable results in S5 is as follows:

[0132] The process rule diagram for generating the image to be detected based on rule constraints. ;

[0133] By integrating the first anomaly heatmap, the second anomaly heatmap, the process rule diagram, and the smoothed anomaly heatmap, the final anomaly heatmap is obtained. ,

[0134]

[0135] in, , , , The heatmap fusion weights are the weight coefficients for the first abnormal heatmap, the second abnormal heatmap, the process rule diagram, and the smoothing abnormal heatmap, respectively. Their values ​​range from [0,1], and the sum of all weight coefficients is 1. The value of each weight is determined by the detection mode and requirements. Specifically: in high-speed online detection mode, improve... ,reduce and When in high-precision composite mode, improve and When processing video streams, improve When the second anomalous heatmap is not present, Preferred, , , ,

[0136] The final anomaly heatmap is normalized and a binary defect mask is generated. Connectivity analysis is performed on the binary defect mask to filter out noise regions with too small an area, resulting in one or more defect boxes. Interpretable results are output.

[0137] The interpretable results include the original image of the image to be detected, the final anomaly heatmap, the overlay of the original image and the final anomaly heatmap, the defect bounding box, the final anomaly score, the low-risk threshold, the high-risk threshold, the value box threshold, the final judgment threshold, the detection inference time, and the structured JSON of the above results. The overlay is used by operators to quickly review the anomaly judgment results, and the structured JSON serves as a data standard format for easy reading. For images to be detected that are determined to be anomalous or defective, the block features corresponding to the maximum memory anomaly score are output to explain the source of the anomaly or defect.

[0138] S6: Based on interpretable results, manually submit missed images found in normal images and false alarm images found in abnormal defect images. Missed images are added to defect samples and the lightweight real-time backbone inference model is updated. False alarm images are added to normal samples and the normal image block memory is updated.

[0139] Furthermore, such as Figure 3 As shown, the specific method of S6 is as follows:

[0140] The operator reviews the interpretable results. When an abnormal defect is found in the normal image that has been judged and output, the image is treated as a missed image and a corresponding defect mask is generated and added to the updated defect sample set. Based on the updated defect sample set, the prediction head is fine-tuned, the lightweight real-time backbone inference model is updated, and the final judgment threshold is lowered first. If necessary, the low-risk threshold is lowered to include more images to be detected in the images to be finely inspected.

[0141] When it is found that the abnormal defect image output by the judgment does not have an abnormal defect, check whether the image is consistent with the normal sample image. If it is consistent, treat it as a false alarm image, add it to the normal sample set, update the normal image block memory, and prioritize raising the final judgment threshold. If necessary, raise the value frame threshold to reduce the interference of background texture response.

[0142] After accumulating multiple feedback submissions from operators, the high-risk threshold can be updated slightly.

[0143] By utilizing interpretable results, operators can be provided with intuitive and comprehensive explanations of defect sources, which can improve the efficiency of manual review. At the same time, by providing feedback on missed and false positive images, the lightweight real-time backbone inference model, normal image block memory, and various thresholds can be continuously updated online, forming a feedback loop and enhancing the self-learning ability of the detection method. This improves the detection method's adaptability to new defects, new workstations, and new textures.

[0144] This invention also provides an optical detection system based on dual-path fusion and feedback self-learning. The optical detection method based on dual-path fusion and feedback self-learning is applied as follows: Figure 4 As shown, it includes:

[0145] Image acquisition and processing module: used to acquire images or video frames of the detection area in real time as images to be detected; preprocessing normal sample images, defect sample images, and images to be detected;

[0146] Lightweight inference module: used to train a lightweight real-time backbone inference model using defect sample images; receives preprocessed images to be detected and outputs the first anomaly heatmap and the first anomaly score;

[0147] Risk gating scheduling module: used to receive the first anomaly score, determine whether the image to be detected is a low-risk image, a medium-risk image or a high-risk image, and output normal images and images to be further inspected;

[0148] The detailed inspection reasoning module is used to receive the image to be inspected, utilize the normal image block memory library, and output the second anomaly heatmap and the second anomaly score.

[0149] Dual-path fusion module: used to receive the first abnormal heatmap, the first abnormal score, the second abnormal heatmap and the second abnormal score, determine the process rule score, the semantic prior score, the video frame stability score and the process rule map, smooth the abnormal heatmap, output normal image or abnormal defect image, and output interpretable results;

[0150] Optical automated inspection platform: such as Figure 5 As shown, it is used to receive and display normal images, abnormal defect images, and interpretable results, and to provide feedback on missed images and false alarm images found by the operator, so as to update the lightweight real-time backbone inference model, normal image block memory, and various thresholds.

[0151] The optical inspection system based on dual-path fusion and feedback self-learning integrates a fast inference path using a lightweight real-time backbone inference model with a fine-tuning inference path using a normal image block memory. Under limited sample conditions, the lightweight inference module performs rapid inspection of the image to be inspected. A risk-gated scheduling module selects medium-risk or high-risk images for the fine-tuning inference module. The dual-path fusion module then generates the final defect judgment result, ensuring both inspection efficiency and accuracy. Operators can use the automatic optical inspection platform to promptly report missed and false alarm images. The system updates itself based on this feedback, achieving adaptive optimization and improving its adaptability to new defects, workstations, and textures, making it more suitable for application in real-world industrial scenarios.

[0152] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An optical detection method based on dual-path fusion and feedback self-learning, characterized in that, Includes the following steps: S1: Collect normal sample images and defective sample images. Based on the defective sample images, train a lightweight real-time backbone inference model. Based on the normal sample images, construct a normal image block memory. S2: Collect images or video frames as images to be detected, preprocess them, input them into a lightweight real-time backbone inference model, and output the corresponding first anomaly heatmap and first anomaly score; S3: Based on the first anomaly score, a risk-gated scheduling method is used to determine whether the image to be detected is a low-risk image, a medium-risk image, or a high-risk image; for low-risk images, they are output as normal images; for medium-risk images, suspicious sub-regions are extracted as images to be further examined; for high-risk images, the entire image is retained as images to be further examined. S4: Based on the normal image block memory, output the second anomaly heatmap and the second anomaly score corresponding to the image to be examined; S5: Introduce a video frame stability detection method, combine the first anomaly score and the second anomaly score, output a normal image or an abnormal defect image, fuse the first anomaly heatmap and the second anomaly heatmap, and output an interpretable result; S6: Based on interpretable results, manually submit missed images found in normal images and false alarm images found in abnormal defect images. Missed images are added to defect samples and the lightweight real-time backbone inference model is updated. False alarm images are added to normal samples and the normal image block memory is updated.

2. The optical detection method based on dual-path fusion and feedback self-learning according to claim 1, characterized in that, The specific method for training the lightweight real-time backbone inference model in S1 is as follows: Before a new product, new workstation, or new inspection task is launched, a small number of defect sample images are collected. Generate the corresponding defect mask for the defect sample image, and add the defect sample image and its corresponding defect mask to the defect sample set; A lightweight real-time backbone inference model is constructed, including a convolutional encoder, a feature fusion layer, and a prediction head. The defect sample image is used as the input image during training, and the corresponding defect mask of the defect sample image is used as the constraint. The loss function is constructed by pixel classification loss and region overlap loss, and the lightweight real-time backbone inference model is trained.

3. The optical detection method based on dual-path fusion and feedback self-learning according to claim 2, characterized in that, The specific method for constructing the normal image block memory in S1 is as follows: Before a new product, new workstation, or new testing task is launched, a small number of normal sample images are collected and added to the normal sample set. For each normal sample image: a pre-defined convolutional neural network is used to extract its multi-layer feature map, and the multi-layer feature map is divided into multiple feature blocks according to its spatial location; Normalize and compress all block features to obtain normal block features, and save the normal block features to the normal image block memory.

4. The optical detection method based on dual-path fusion and feedback self-learning according to claim 3, characterized in that, The specific method for outputting the first anomaly heatmap and the first anomaly score in S2 is as follows: input the input image to be detected into the lightweight real-time backbone inference model to generate an anomaly response map corresponding to the input image to be detected, and use it as the first anomaly heatmap. The maximum value of the abnormal response in the first abnormal heatmap is selected as the first abnormal score. , that is, Where p represents the pixel position in the first anomaly heatmap. This represents the entire first anomaly heatmap region.

5. The optical detection method based on dual-path fusion and feedback self-learning according to claim 4, characterized in that, The specific method for determining low-risk, medium-risk, or high-risk images in S3 is as follows: Obtain the first anomaly score of the image to be detected. Set a low-risk threshold and high risk threshold , ; like If so, the corresponding image to be detected is determined to be a low-risk image; like If so, the corresponding image to be detected is determined to be a medium-risk image; like If the image is detected, it is determined to be a high-risk image.

6. The optical detection method based on dual-path fusion and feedback self-learning according to claim 5, characterized in that, The specific method of S4 is as follows: Multiple feature blocks to be refined are extracted from each image to be refined. Calculate the Euclidean distance between each feature of the block to be refined and each normal block feature in the normal image block memory. The minimum Euclidean distance is used to obtain the memory anomaly score for each feature of the block to be refined, i.e.: in, , Let be the memory anomaly score and feature value of the i-th feature block to be refined, respectively. Let i be the feature value of the j-th normal block feature, i = 1, 2, 3, ..., j = 1, 2, 3, ...; The anomaly scores of each feature block to be refined are used to generate a second anomaly heatmap based on their spatial location in the image to be refined. , ; Extract the second anomaly score from the second anomaly heatmap. , in, This refers to the pixel location in the second anomaly heatmap. This represents the entire area of ​​the second anomaly heatmap.

7. The optical detection method based on dual-path fusion and feedback self-learning according to claim 6, characterized in that, The video frame stability detection method in S5 is specifically as follows: For the image to be detected originating from the t-th interval frame, the first and second anomaly heatmaps are fused to generate a fused anomaly heatmap. The fused anomaly heatmap is then processed using an exponential moving average to obtain a smoothed anomaly heatmap, i.e. Where t is a positive integer, This is a smoothed anomaly heatmap of the image to be detected. This is a smoothed anomaly heatmap of the previous frame in the image to be detected. This is the smoothing coefficient, and its value range is (0,1). Obtain the anomaly detection results of the image to be detected and its n adjacent interval frames, perform short-window voting, and obtain the smoothed voting score of the image to be detected. , Where n is a positive integer ranging from [3, 5]. The result of anomaly detection is given for k adjacent frames of the image to be detected. The maximum response value in the smooth anomaly heatmap is selected as the smooth anomaly score. The video frame stability score is obtained by combining the smoothed voting score. , .

8. The optical detection method based on dual-path fusion and feedback self-learning according to claim 7, characterized in that, The specific method for outputting normal or abnormal defect images in S5 is as follows: Introducing process rule scores Semantic prior scores The final anomaly score is calculated by combining the first anomaly score, the second anomaly score, and the video frame stability score. , in, , , , , These are the first weight, the second weight, the process rule weight, the video frame stabilization weight, and the semantic prior weight, respectively. Their values ​​are all in the range of [0,1], and the sum of their weights is 1. The value of each weight is determined by the detection mode and requirements. Set the final judgment threshold ,when When the image to be detected is determined to be a normal image, it is output. When the image to be detected is determined to be an abnormal defect image and output, the final determination threshold value range is (0,1).

9. The optical detection method based on dual-path fusion and feedback self-learning according to claim 8, characterized in that, The specific methods for interpreting the S6 types of output results are as follows: By integrating the first anomaly heatmap, the second anomaly heatmap, the process rule diagram, and the smoothed anomaly heatmap, the final anomaly heatmap is obtained. , in, , , , The heatmap fusion weights are the weight coefficients of the first abnormal heatmap, the second abnormal heatmap, the process rule graph, and the smooth abnormal heatmap, respectively. Their values ​​are all in the range of [0,1], and the sum of each weight coefficient is 1. The process rule graph is obtained by constraining the image to be detected according to the process rules. Defect boxes are obtained from the final anomaly heatmap, and interpretable results are output, including the original image of the image to be detected, the final anomaly heatmap, the overlay of the original image and the final anomaly heatmap, the defect boxes, the final anomaly score, the low-risk threshold, the high-risk threshold, the final judgment threshold, the detection inference time, and the structured JSON of the above results.

10. An optical detection system based on dual-path fusion and feedback self-learning, applied to the optical detection method based on dual-path fusion and feedback self-learning as described in any one of claims 1-9, characterized in that, include: Lightweight inference module: used to train a lightweight real-time backbone inference model using defect sample images, receive preprocessed images to be detected, and output the first anomaly heatmap and the first anomaly score; Risk gating scheduling module: used to receive the first anomaly score, determine whether the image to be detected is a low-risk image, a medium-risk image or a high-risk image, and output normal images and images to be further inspected; The detailed inspection reasoning module is used to receive the image to be inspected, utilize the normal image block memory library, and output the second anomaly heatmap and the second anomaly score. Dual-path fusion module: used to receive the first abnormal heatmap, the first abnormal score, the second abnormal heatmap and the second abnormal score, and output a normal image or an abnormal defect image, and output interpretable results; Optical automated inspection platform: used to receive and display normal images, abnormal defect images, and interpretable results; to provide feedback on missed and false alarm images found by the operator; and to update the lightweight real-time backbone inference model, normal image block memory, and various thresholds.

Images