Multi-branch self-learning defect detection method and device, equipment and storage medium

Abstract

The invention discloses a multi-branch self-learning defect detection method, device and equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: collecting a plurality of frames of images in a continuous movement process of a workpiece; performing motion compensation and illumination correction on the multiple frames of images to obtain aligned multiple frames of images; constructing a time sequence consistency tensor for target pixel neighborhoods in the aligned multiple frames of images; based on the time sequence consistency tensor, outputting a defect consistency score and a model uncertainty score of a target pixel neighborhood through a double-branch network; performing weighted summation processing on the defect consistency score and the model uncertainty score according to a preset weight coefficient to obtain a final defect judgment score; and comparing the final defect judgment score with a dynamic defect judgment threshold, and determining whether the target pixel neighborhood is a defect area, so that interference factors such as jitter and illumination variation can be effectively dealt with in a complex production environment, and the accuracy and real-time performance of detection are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-branch self-learning defect detection method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of automated production technology, rapid and accurate inspection of product appearance defects on the production line has become a key link in improving product quality and production efficiency. Traditional manual inspection methods are not only inefficient but also greatly affected by subjective factors, making it difficult to meet the needs of high-speed production lines. Automated defect detection technology has emerged to address this need. Its core objective is to use technologies such as machine vision to detect product appearance defects in real time, promptly identify quality problems in the production process, and thus achieve automation and intelligence in quality control.

[0003] Existing automated defect detection technologies mainly fall into two categories. One category is defect detection methods based on frame difference and optical flow. These methods primarily utilize image differences or optical flow estimation between consecutive frames to capture motion information and thus detect defects. However, these methods are prone to high false alarm and false negative rates when faced with complex environmental factors such as equipment vibration, lighting changes, and reflections in high-speed production lines, and their performance in detecting static defects is poor. The other category is defect detection methods based on convolutional neural networks (CNNs) and object detection algorithms. These methods require a large amount of labeled defect data for training and are highly dependent on data. In practical industrial applications, acquiring large amounts of labeled data is often costly and time-consuming. Furthermore, CNNs have poor robustness to lighting changes, noise, and complex backgrounds, easily leading to false alarms and making them difficult to adapt to dynamic and complex production environments.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a multi-branch self-learning defect detection method, apparatus, device, and storage medium, aiming to solve the technical problems of inaccurate defect detection and poor robustness in complex dynamic environments.

[0006] To achieve the above objectives, the present invention provides a multi-branch self-learning defect detection method, which includes the following steps:

[0007] Acquire multiple frames of images during the continuous movement of the workpiece;

[0008] Motion compensation and illumination correction are performed on the multi-frame images to obtain aligned multi-frame images;

[0009] Construct a temporally consistent tensor for the neighborhood of the target pixels in the aligned multi-frame images;

[0010] Based on the temporal consistency tensor, the defect consistency score and model uncertainty score of the target pixel neighborhood are output through a dual-branch network.

[0011] The final defect determination score is obtained by weighting and summing the defect consistency score and the model uncertainty score according to preset weighting coefficients.

[0012] The final defect determination score is compared with the dynamic defect determination threshold to determine whether the neighborhood of the target pixel is a defect region.

[0013] In one embodiment, the step of performing motion compensation and illumination correction on the multi-frame images to obtain aligned multi-frame images includes:

[0014] The multi-frame images are analyzed by a motion and illumination alignment learning network to obtain the predicted relative displacement parameters and illumination correction parameters between adjacent frames.

[0015] The displacement compensation of the multi-frame images is performed using the relative displacement parameters to obtain a displacement-corrected image;

[0016] The illumination is adjusted on the displacement-corrected image using the illumination correction parameters to obtain aligned multi-frame images.

[0017] In one embodiment, the step of constructing a temporally consistent tensor for the neighborhood of target pixels in the aligned multi-frame images includes:

[0018] In the aligned multi-frame images, a fixed-size neighborhood centered on the target pixel is selected as the target pixel neighborhood;

[0019] Calculate the average brightness value of the target pixel neighborhood in the aligned multi-frame images to obtain the average brightness value;

[0020] The brightness variance is obtained by calculating the brightness variation of the target pixel neighborhood in the aligned multi-frame images.

[0021] The brightness correlation of the target pixel neighborhood in the current frame image and the previous frame image is calculated to obtain the inter-frame covariance;

[0022] The temporal consistency tensor is obtained based on the mean brightness, the variance of brightness, and the inter-frame covariance.

[0023] In one embodiment, the step of outputting the defect consistency score and model uncertainty score of the target pixel neighborhood through a dual-branch network based on the temporal consistency tensor includes:

[0024] Features of the temporal consistency tensor are extracted using a convolutional neural network;

[0025] The features of the temporal consistency tensor are processed by the consistency branch of the dual-branch network to obtain the defect consistency score of the target pixel neighborhood.

[0026] The features of the temporal consistency tensor are processed by the uncertainty branch of the dual-branch network to obtain the model uncertainty score of the target pixel neighborhood.

[0027] In one embodiment, the step of comparing the final defect determination score with a dynamic defect determination threshold to determine whether the target pixel neighborhood is a defect region includes:

[0028] The final defect determination score is compared with the dynamic defect determination threshold to obtain the comparison result;

[0029] When the comparison result is that the final defect determination score is greater than the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a defect region;

[0030] When the comparison result is that the final defect determination score is less than or equal to the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a non-defect region.

[0031] In one embodiment, after the step of comparing the final defect determination score with the dynamic defect determination threshold to obtain the comparison result, the method further includes:

[0032] Obtain the false alarm rate and false negative rate data of the comparison results;

[0033] The false positive rate and false negative rate data are input into the reinforcement learning model to obtain the detection results;

[0034] Based on the detection results, the dynamic defect determination threshold is adjusted and updated.

[0035] In one embodiment, the step of weighting and summing the defect consistency score and the model uncertainty score according to preset weighting coefficients to obtain the final defect judgment score includes:

[0036] The defect consistency score and the model uncertainty score are weighted and summed according to preset weighting coefficients to obtain the normalized final defect judgment score.

[0037] The normalized final defect determination score is linearly transformed and mapped to a preset determination score range to obtain the target defect determination score.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a multi-branch self-learning defect detection device, the device comprising:

[0039] The acquisition module is used to acquire multiple frames of images during the continuous movement of the workpiece;

[0040] The image processing module is used to perform motion compensation and illumination correction on the multi-frame images to obtain aligned multi-frame images; and to construct a temporal consistency tensor for the neighborhood of the target pixels in the aligned multi-frame images.

[0041] The scoring module is used to output the defect consistency score and model uncertainty score of the target pixel neighborhood through a dual-branch network based on the temporal consistency tensor; and to obtain the final defect judgment score by weighted summation of the defect consistency score and the model uncertainty score according to preset weight coefficients.

[0042] The judgment module is used to compare the final defect judgment score with the dynamic defect judgment threshold to determine whether the target pixel neighborhood is a defect region.

[0043] Furthermore, to achieve the above objectives, the present invention also proposes a multi-branch self-learning defect detection device, the device comprising: a memory, a processor, and a multi-branch self-learning defect detection program stored in the memory and executable on the processor, the multi-branch self-learning defect detection program being configured to implement the steps of the multi-branch self-learning defect detection method as described above.

[0044] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a multi-branch self-learning defect detection program, which, when executed by a processor, implements the steps of the multi-branch self-learning defect detection method described above.

[0045] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multi-branch self-learning defect detection method described above.

[0046] One or more technical solutions proposed in this application have at least the following technical effects:

[0047] Multiple frames of images are acquired during the continuous movement of the workpiece; motion compensation and illumination correction are performed on the multiple frames of images to obtain aligned multiple frames of images; a temporal consistency tensor is constructed for the neighborhood of the target pixel in the aligned multiple frames of images; based on the temporal consistency tensor, a dual-branch network is used to output the defect consistency score and model uncertainty score of the neighborhood of the target pixel; the defect consistency score and model uncertainty score are weighted and summed according to preset weight coefficients to obtain the final defect judgment score; the final defect judgment score is compared with the dynamic defect judgment threshold to determine whether the neighborhood of the target pixel is a defect area. This method can effectively cope with interference factors such as jitter and illumination changes in complex production environments, and improve the accuracy and real-time performance of detection. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an embodiment of the multi-branch self-learning defect detection method of this application.

[0051] Figure 2 This is a flowchart illustrating Embodiment 2 of the multi-branch self-learning defect detection method of this application;

[0052] Figure 3 This is a schematic diagram of the module structure of the multi-branch self-learning defect detection device according to an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-branch self-learning defect detection method in this application embodiment.

[0054] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0057] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a multi-branch self-learning defect detection device. The following description uses a multi-branch self-learning defect detection device as an example to illustrate this embodiment and the subsequent embodiments.

[0058] Based on this, embodiments of this application provide a multi-branch self-learning defect detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-branch self-learning defect detection method of this application.

[0059] In this embodiment, the multi-branch self-learning defect detection method includes steps S10 to S60:

[0060] Step S10: Acquire multiple frames of images during the continuous movement of the workpiece;

[0061] It should be noted that in automated production lines, workpieces are typically in continuous motion. To automate the detection of surface defects, multiple frames of images need to be captured continuously during the workpiece's movement. These images serve as the basis for subsequent processing and analysis, capturing detailed information about the workpiece surface to detect areas where defects may exist.

[0062] It should be understood that the TC-Tensor (Temporal-Consistency Tensor) refers to the calculation of the mean, variance, and inter-frame covariance of the same pixel neighborhood in a series of consecutive frames of images, which is used to evaluate the stability of image regions.

[0063] MOAL-Net (Motion-&Illumination-Aligned Learning Network) is a motion and illumination alignment learning network that combines displacement compensation and illumination correction to correct deviations in images caused by motion or illumination changes.

[0064] UD-Branch (Uncertainty Dual-branch) refers to the second branch of the network outputting uncertainty (Aleatoric and Epistemic), which measures the network's confidence in the current input.

[0065] RL-Thre (Reinforcement-Learning Threshold Controller) is a reinforcement learning threshold controller that dynamically adjusts the defect detection threshold based on reinforcement learning algorithms to minimize false alarms and missed detections.

[0066] MV-Check (Multi-View Consistency Check) refers to the mutual verification and review of defect detection results by comparing images from multiple perspectives at upstream and downstream workstations.

[0067] In practical implementation, a high-resolution line scan camera or area scan camera is used to capture one frame of image at a time. The acquisition time and workpiece position information of each image are synchronously recorded via an external incremental encoder and trigger. This precise synchronization of the industrial camera and encoder ensures that multiple consecutive frames are captured for each workpiece within the same time period.

[0068] The acquired image frames are buffered and arranged in order. A fixed sliding window is set, and N consecutive frames are analyzed at a time to capture the temporal changes of defects between consecutive frames.

[0069] Step S20: Perform motion compensation and illumination correction on the multi-frame images to obtain aligned multi-frame images;

[0070] It should be noted that because the workpiece is in motion during image acquisition, there are displacement differences between the acquired multiple frames. Motion compensation technology analyzes the relative displacement between adjacent frames to perform translation correction on the images, aligning multiple frames in space and ensuring that pixels at the same location correspond to the same physical position in subsequent analysis.

[0071] Changes in lighting conditions can affect the brightness and contrast of an image, thus impacting the accuracy of defect detection. Lighting correction adjusts the brightness of the image to ensure consistent lighting conditions across multiple frames, reducing the interference of lighting variations on the detection results.

[0072] It should be understood that this step proposes the Motion-Light Coupled Alignment Network (MOAL-Net), an end-to-end alignment network that combines displacement compensation and illumination correction. When dealing with motion or illumination variations in images, MOAL-Net effectively eliminates the effects of camera shake and unstable lighting, ensuring image alignment accuracy. Through displacement compensation and gamma-polarized illumination correction, it maintains high detection accuracy in dynamic and complex production environments.

[0073] Understandably, this application incorporates several innovations, including Temporally Consistent Tensor (TC-Tensor), Motion-Light Coupled Alignment Network (MOAL-Net), Uncertainty Bi-Branch Decision Mechanism (UD-Branch), and Reinforcement Learning Dynamic Thresholding (RL-thre). The core objective is to automatically detect surface defects on industrial production lines with low data requirements, high accuracy, and high robustness, especially under dynamic environments and unstable lighting conditions.

[0074] In practical implementation, due to the high-speed movement and changing lighting conditions on the production line, images may contain errors caused by camera shake, lighting changes, or reflections. To address this issue, MOAL-Net is proposed. This network can perform displacement compensation and lighting correction on images; it uses optical flow and sub-pixel phase correlation methods to estimate the relative displacement between adjacent frames; and it performs uniform correction on images under different lighting conditions through gamma correction and a polarization intensity model to reduce the interference of lighting changes.

[0075] The structure of the motion-lighting coupled alignment network includes:

[0076] Displacement branch: Image displacement information is extracted through multiple convolutional layers (3×3), and then displacements Δx and Δy are predicted through fully connected layers (FC).

[0077] Illumination branch: Extract the reflection texture information of the image through a convolutional neural network (CNN), and then regress the illumination parameters (gamma γ and polarization coefficient p).

[0078] Output: The network finally outputs the corrected image (I_t') and displacement and illumination parameters.

[0079] The image alignment process uses the aforementioned displacement and illumination correction parameters to preprocess the input image, ensuring that consecutive frames are compared and analyzed under the same standard.

[0080] In one feasible implementation, step S20 includes steps A11 to A13:

[0081] A11: By analyzing multiple frames of images through a motion and illumination alignment learning network, the predicted relative displacement parameters and illumination correction parameters between adjacent frames are obtained;

[0082] It should be noted that the motion and illumination alignment learning network is a deep learning model used to analyze image sequences. It predicts relative displacement parameters and illumination correction parameters between adjacent frames by learning motion and illumination variation patterns in the image sequence. The purpose of this step is to provide the necessary parameter support for subsequent motion compensation and illumination correction.

[0083] A12: Displacement compensation is performed on multiple frames of images using relative displacement parameters to obtain displacement-corrected images;

[0084] It should be noted that displacement compensation is performed on multiple frames of images using predicted relative displacement parameters. Displacement compensation involves translating pixels in the image according to the predicted displacement amount, thereby aligning the multiple frames in space. The purpose of this step is to eliminate image displacement differences caused by workpiece movement, ensuring the accuracy of subsequent processing.

[0085] A13: Adjust the illumination of the displacement-corrected image using illumination correction parameters to obtain aligned multi-frame images.

[0086] It should be noted that the predicted illumination correction parameters are used to adjust the illumination of the displacement-corrected image. Illumination adjustment involves adjusting the brightness and contrast of the image to make the illumination conditions of multiple frames more consistent. The purpose of this step is to reduce the interference of illumination changes on the detection results and improve the robustness of the detection.

[0087] Understandably, this solution can use feature-point-based alignment methods, such as SIFT (Scale Invariant Feature Transform) or SURF (Speed-Up Robust Feature Transform), to perform image alignment as a replacement for the Motion-Light Alignment Network (MOAL-Net). The specific steps include:

[0088] Extract salient feature points (such as corners and edges) from the image and use SIFT or SURF algorithms to extract key points; infer the relative displacement and rotation transformations between adjacent frames by calculating feature matching between images; align adjacent frames to enable comparative analysis of defects on images with inconsistent relative positions.

[0089] Step S30: Construct a temporally consistent tensor for the neighborhood of the target pixel in the aligned multi-frame images;

[0090] It should be noted that in the aligned multi-frame images, a fixed-size neighborhood is selected centered on the target pixel. This neighborhood is chosen to focus on local areas in the image that may contain defects, allowing for more detailed analysis.

[0091] A ternary tensor is constructed by calculating the mean brightness, variance brightness, and inter-frame covariance of the target pixel's neighborhood across consecutive frames. This tensor characterizes the brightness stability of the target pixel's neighborhood over time, providing crucial features for subsequent defect detection.

[0092] It should be understood that by introducing the Temporal Consistency Tensor (TC-Tensor) and combining the image mean, variance, and covariance between consecutive frames, a novel approach is proposed to evaluate the stability of image regions and thus detect defects. Compared to traditional frame difference or optical flow methods, the TC-Tensor can not only more accurately capture static defects (such as scratches, dents, and blemishes), but also effectively resist false alarms caused by factors such as changes in lighting and device jitter.

[0093] In the specific implementation, neighborhood selection is performed as follows: for each frame of the image, a neighborhood of size k×k is selected, and the features of that neighborhood are extracted in N consecutive frames.

[0094] For each pixel (u,v), calculate its mean (μ) and variance (σ) in its neighborhood over N frames.

[0095] 2) The covariance (τ) forms a ternary tensor.

[0096] Where μ represents the average brightness of the pixel over N frames, reflecting the stability of the pixel.

[0097] σ2 represents the degree of brightness variation of the pixel and is used to detect the fluctuation of the area.

[0098] τ represents the covariance between the current frame and the previous frame, reflecting the similarity of the pixel in consecutive frames.

[0099] The computation results are combined into a temporally consistent tensor, the TC-Tensor structure being {μ,σ}. 2 The derivation is performed on the derivation of τ and stored as a three-dimensional tensor. This tensor will be used for subsequent defect detection and classification.

[0100] By comparing the temporal consistency tensors of image regions, consistency differences in defective regions can be identified. Defects typically lead to increased brightness fluctuations within the region (σ). 2 (Increases), and the relationship with the preceding and following frames becomes inconsistent (τ decreases).

[0101] Understandably, this scheme can use frame difference or optical flow to calculate the changes between image frames, and combine it with sliding window-based statistical analysis to extract the features of regional changes to replace the temporal consistency tensor (TC-Tensor). The specific steps include: calculating the frame difference of pixels through consecutive image frames to obtain the brightness changes between adjacent frames; using optical flow to calculate the displacement of moving objects and identifying possible defects through motion analysis; and performing sliding window processing on the frame difference or optical flow results of the image region to calculate the change features of the region and further infer defects.

[0102] In one feasible implementation, step S30 includes steps A21 to A25:

[0103] A21: Select a fixed-size neighborhood centered on the target pixel as the target pixel neighborhood in the aligned multi-frame images;

[0104] It should be noted that, in the aligned multi-frame images, a fixed-size neighborhood centered on the target pixel is selected. The size of this neighborhood is typically determined based on the actual detection requirements and the typical size of the defect, with the aim of ensuring that the selected region contains sufficient information for defect detection.

[0105] A22: Calculate the average brightness value of the target pixel's neighborhood in the aligned multi-frame image to obtain the average brightness value;

[0106] It should be noted that the average brightness value is obtained by averaging the brightness values ​​of the target pixel's neighborhood across multiple frames of images. The average brightness value reflects the average brightness level of the region over time and is an important indicator for evaluating the brightness stability of a region.

[0107] A23: Calculate the brightness variation of the target pixel's neighborhood in the aligned multi-frame images to obtain the brightness variance;

[0108] It should be noted that the brightness variance is obtained by calculating the brightness variation of the target pixel's neighborhood across multiple frames of images. The brightness variance reflects the fluctuation of the brightness value in that area; a larger variance may indicate the presence of defects or interference in that area.

[0109] A24: Calculate the brightness correlation of the target pixel's neighborhood in the current frame image and the previous frame image to obtain the inter-frame covariance;

[0110] It should be noted that the inter-frame covariance is obtained by calculating the brightness correlation of the target pixel's neighborhood in the current frame and the previous frame. The inter-frame covariance reflects the correlation of brightness changes between adjacent frames and can be used to evaluate the stability of image sequences.

[0111] A25: Obtain the temporal consistency tensor based on the mean luminance, luminance variance, and inter-frame covariance.

[0112] It should be noted that integrating the mean brightness, variance brightness, and inter-frame covariance into a single ternary tensor results in the temporal consistency tensor. This tensor characterizes the brightness properties of the target pixel's neighborhood over time from multiple dimensions, providing a comprehensive feature description for subsequent defect detection.

[0113] Step S40: Based on the temporal consistency tensor, output the defect consistency score and model uncertainty score of the target pixel neighborhood through a dual-branch network;

[0114] It should be noted that by using a convolutional neural network to extract features from the temporal consistency tensor, the extracted features can reflect the brightness change characteristics of the target pixel neighborhood over time.

[0115] The dual-branch network consists of a consistency branch and an uncertainty branch. The consistency branch calculates a defect consistency score for the neighborhood of the target pixel based on the extracted features. This score reflects the probability that a defect exists in the target region. The uncertainty branch calculates the model uncertainty score, which is used to assess the model's confidence in the current input.

[0116] It should be understood that this application not only outputs a defect consistency score, but also estimates the model's uncertainty (Aleatoric and Epistemic). This dual-output mechanism enables the defect detection system to reduce the false alarm rate under high uncertainty conditions, while improving the system's reliability and interpretability.

[0117] In the specific implementation, two sub-branches are used to calculate Aleatoric (uncertainty caused by data noise) and Epistemic (model uncertainty), respectively. By calculating the variance of these two variables, the model's confidence in the current input image can be evaluated.

[0118] Aleatoric uncertainty, calculated for each pixel using a regression model, represents the instability caused by image noise.

[0119] Epistemic uncertainty is calculated by using the Monte Carlo Dropout method to perform multiple forward inferences and calculate the variance of the uncertainty.

[0120] In one feasible implementation, step S40 includes steps A31 to A33:

[0121] A31: Extracting features of temporally consistent tensors using convolutional neural networks;

[0122] It should be noted that Convolutional Neural Networks (CNNs) are a type of deep learning model that excels at processing image data. Through convolution and pooling operations, CNNs can extract discriminative features from temporal consistency tensors. These features can capture the brightness variation patterns of the target pixel's neighborhood over time, providing a basis for subsequent defect determination.

[0123] A32: The features of the temporal consistency tensor are processed by the consistency branch of the dual-branch network to obtain the defect consistency score of the target pixel neighborhood.

[0124] It's important to note that the consistency branch is a branch of the two-branch network, focusing on evaluating the likelihood of defects existing in the neighborhood of the target pixel. By processing the extracted features, the consistency branch outputs a score, known as the defect consistency score. The higher this score, the greater the likelihood that the target region contains defects.

[0125] A33: By processing the features of the temporal consistency tensor through the uncertainty branch of the dual-branch network, the model uncertainty score of the target pixel neighborhood is obtained.

[0126] It should be noted that the uncertainty branch is another branch of the two-branch network, primarily used to evaluate the model's confidence level in the current input. By processing the extracted features, the uncertainty branch outputs a score, namely the model uncertainty score. This score reflects the model's uncertainty about the existence of defects in the target region and can be used to assist in judging the reliability of the detection results.

[0127] Step S50: The defect consistency score and model uncertainty score are weighted and summed according to preset weighting coefficients to obtain the final defect judgment score.

[0128] It should be noted that the defect consistency score and the model uncertainty score are weighted and summed according to preset weighting coefficients. The weighting coefficients are set to balance the influence of the two scores on the final judgment result, so that the final defect judgment score can comprehensively reflect the probability of whether there is a defect in the neighborhood of the target pixel and the model's confidence in this judgment.

[0129] In the specific implementation, the consistency score and uncertainty score are weighted and combined to obtain the final judgment score S. i This is used for subsequent defect decision-making.

[0130] S i =α(1-C i )+βU i

[0131] Here, α and β are weighting coefficients that control the impact of consistency score and uncertainty on the final decision.

[0132] In one feasible implementation, step S50 includes steps A41 to A42:

[0133] A41: The defect consistency score and the model uncertainty score are weighted and summed according to the preset weighting coefficients to obtain the normalized final defect judgment score.

[0134] It should be noted that the defect consistency score and model uncertainty score are weighted and summed according to preset weighting coefficients, and the result is normalized to a preset interval. Normalization ensures that scores from different ranges are unified to the same scale, facilitating subsequent comparison and judgment. If S... i If the value exceeds the set threshold, the pixel is marked as a defect area.

[0135] A42: Perform a linear transformation on the normalized final defect judgment score to map it to a preset judgment score interval to obtain the target defect judgment score.

[0136] It should be noted that a linear transformation is performed on the normalized final defect judgment score to map it to a preset judgment score range. The purpose of this step is to ensure that the score falls within a suitable range for threshold comparison, facilitating subsequent defect judgment.

[0137] Step S60: Compare the final defect determination score with the dynamic defect determination threshold to determine whether the neighborhood of the target pixel is a defect area.

[0138] It should be noted that the final defect determination score is compared with the dynamic defect determination threshold. If the final defect determination score is greater than the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a defective region; otherwise, the neighborhood of the target pixel is determined to be a non-defective region. The dynamic defect determination threshold is adjusted in real time based on the false alarm rate and false negative rate during the detection process to optimize detection performance.

[0139] This embodiment provides a multi-branch self-learning defect detection method. Multiple frames of images are acquired during continuous workpiece movement. Motion compensation and illumination correction are performed on the multiple frames to obtain aligned images. A temporal consistency tensor is constructed for the neighborhood of the target pixel in the aligned images. Based on the temporal consistency tensor, a dual-branch network outputs the defect consistency score and model uncertainty score of the target pixel neighborhood. The defect consistency score and model uncertainty score are weighted and summed according to preset weight coefficients to obtain the final defect judgment score. The final defect judgment score is compared with a dynamic defect judgment threshold to determine whether the target pixel neighborhood is a defect area. This method can effectively cope with interference factors such as jitter and illumination changes in complex production environments, improving the accuracy and real-time performance of detection.

[0140] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S60 includes steps S601 to S603:

[0141] Step S601: Compare the final defect judgment score with the dynamic defect judgment threshold to obtain the comparison result;

[0142] It should be noted that performing a numerical comparison operation, comparing the final defect determination score with the current dynamic defect determination threshold, aims to determine the relationship between the score and the threshold, thereby providing a basis for subsequent determination of whether the target pixel neighborhood is a defect area.

[0143] Step S602: When the comparison result is that the final defect judgment score is greater than the dynamic defect judgment threshold, the neighborhood of the target pixel is determined as a defect region.

[0144] It should be noted that when the comparison results show that the final defect determination score exceeds the dynamic defect determination threshold, it means that according to the model's evaluation, the neighborhood of the target pixel is likely to have a defect, which meets the criteria for being determined as a defect, and therefore it is identified as a defect area.

[0145] Step S603: When the comparison result is that the final defect judgment score is less than or equal to the dynamic defect judgment threshold, the neighborhood of the target pixel is determined to be a non-defect region.

[0146] It should be noted that if the comparison results show that the final defect judgment score does not exceed the dynamic defect judgment threshold, it is considered that there are no obvious defects in the neighborhood of the target pixel, or that the existing defects are within an acceptable range, and therefore it is judged as a non-defect area.

[0147] In one feasible implementation, after step S603, steps A11 to A13 are further included:

[0148] A11: Obtain the false alarm rate and false negative rate data of the comparison results;

[0149] It should be noted that after completing a series of judgments, the judgment results are statistically analyzed to calculate the false alarm rate (the proportion of defect-free areas incorrectly identified as defective areas) and the false negative rate (the proportion of defective areas incorrectly identified as non-defective areas). These data reflect the performance of the current detection system.

[0150] A12: Input the false positive rate and false negative rate data into the reinforcement learning model to obtain the detection results;

[0151] It should be noted that the false positive rate and false negative rate data collected above are used as input features for the reinforcement learning model. The reinforcement learning model learns the detection performance reflected by these data, evaluates the effectiveness of the current detection strategy, and outputs corresponding detection results, which include feedback information on the current detection decision.

[0152] A13: Adjust and update the dynamic defect judgment threshold based on the test results.

[0153] It should be noted that the dynamic defect judgment threshold is adjusted based on the detection results output by the reinforcement learning model. If the false alarm rate or false negative rate is too high, the model will indicate that the threshold should be adjusted in the corresponding direction to reduce the probability of false alarms or false negatives in subsequent detections, so that the detection system can better adapt to changes in the actual production environment and optimize detection performance.

[0154] It should be understood that by using reinforcement learning algorithms, the detection threshold is adjusted in real time based on the false alarm rate and the false negative rate to achieve adaptive defect judgment. Unlike traditional static thresholding methods, RL-thre can continuously optimize the threshold during the production process to ensure optimal detection performance under different production environments and workpiece types.

[0155] In the specific implementation, the initial threshold setting is as follows: using the statistical process control (SPC) method, the initial threshold Θ0 = μ + 3σ is set by calculating the mean μ and standard deviation σ of the normal image.

[0156] Furthermore, by using reinforcement learning (Q-Learning) algorithm, the defect judgment threshold is dynamically adjusted based on the current false positive rate (FP) and false negative rate (FN). Through continuous learning and feedback, the reinforcement learning algorithm enables the system to automatically adjust the threshold in different production environments to achieve the best detection effect.

[0157] Furthermore, this strategy also employs Q-Learning updates. After each detection, the reward for threshold adjustment (such as reducing false positives or false negatives) is calculated, and the current threshold Θt is updated via Q-Learning.

[0158] This embodiment provides a multi-branch self-learning defect detection method.

[0159] By dynamically adjusting the defect judgment threshold, this method can effectively reduce the false alarm rate and false negative rate, improve the accuracy of defect detection, and ensure the reliability of product quality on the production line. By using reinforcement learning to continuously optimize the threshold based on actual detection results, the detection system has stronger adaptability, can cope with complex changes in the production environment and different workpiece types, maintain stable detection performance, and realize full-process automation and intelligence from image acquisition, feature extraction, defect judgment to threshold update. This reduces manual intervention, improves production efficiency, and helps to achieve the upgrade of intelligent manufacturing.

[0160] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-branch self-learning defect detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0161] This application also provides a multi-branch self-learning defect detection device; please refer to... Figure 3 The multi-branch self-learning defect detection device includes:

[0162] Acquisition module 10 is used to acquire multiple frames of images during the continuous movement of the workpiece;

[0163] Image processing module 20 is used to perform motion compensation and illumination correction on multiple frames of images to obtain aligned multiple frames of images; and to construct a temporal consistency tensor for the neighborhood of the target pixels in the aligned multiple frames of images.

[0164] The scoring module 30 is used to output the defect consistency score and model uncertainty score of the target pixel neighborhood based on the temporal consistency tensor through a dual-branch network; the defect consistency score and model uncertainty score are weighted and summed according to preset weight coefficients to obtain the final defect judgment score.

[0165] The judgment module 40 is used to compare the final defect judgment score with the dynamic defect judgment threshold to determine whether the neighborhood of the target pixel is a defect area.

[0166] The multi-branch self-learning defect detection device provided in this application, employing the multi-branch self-learning defect detection method described in the above embodiments, can solve the technical problems of inaccurate defect detection and poor robustness in complex dynamic environments. Compared with the prior art, the beneficial effects of the multi-branch self-learning defect detection device provided in this application are the same as those of the multi-branch self-learning defect detection method described in the above embodiments, and other technical features in the multi-branch self-learning defect detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0167] In one embodiment, the image processing module 20 is further configured to analyze multiple frames of images through a motion and illumination alignment learning network to obtain predicted relative displacement parameters and illumination correction parameters between adjacent frames;

[0168] Displacement compensation is performed on multiple frames of images using relative displacement parameters to obtain displacement-corrected images;

[0169] The illumination of the displacement-corrected image is adjusted by using illumination correction parameters to obtain aligned multi-frame images.

[0170] In one embodiment, the image processing module 20 is further configured to select a fixed-size neighborhood centered on the target pixel as the target pixel neighborhood in the aligned multi-frame images;

[0171] Calculate the average brightness value of the target pixel's neighborhood in the aligned multi-frame images to obtain the average brightness value;

[0172] The brightness variance is obtained by calculating the brightness variation of the target pixel's neighborhood in the aligned multi-frame images.

[0173] Calculate the brightness correlation of the target pixel's neighborhood in the current frame image and the previous frame image to obtain the inter-frame covariance;

[0174] Based on the mean luminance, luminance variance, and inter-frame covariance, the temporal consistency tensor is obtained.

[0175] In one embodiment, the scoring module 30 is further configured to extract features of the temporal consistency tensor via a convolutional neural network;

[0176] The features of the temporal consistency tensor are processed by the consistency branch of the dual-branch network to obtain the defect consistency score of the target pixel neighborhood.

[0177] By processing the characteristics of the temporal consistency tensor through the uncertainty branch of the dual-branch network, the model uncertainty score of the target pixel neighborhood is obtained.

[0178] In one embodiment, the scoring module 30 is further configured to compare the final defect determination score with the dynamic defect determination threshold to obtain a comparison result;

[0179] When the comparison result shows that the final defect judgment score is greater than the dynamic defect judgment threshold, the neighborhood of the target pixel is determined as the defect region.

[0180] When the comparison result shows that the final defect determination score is less than or equal to the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a non-defect region.

[0181] In one embodiment, the scoring module 30 is further configured to obtain false alarm rate and false negative rate data of the comparison results;

[0182] The false positive rate and false negative rate data are input into the reinforcement learning model to obtain the detection results;

[0183] Based on the test results, adjust and update the dynamic defect judgment threshold.

[0184] In one embodiment, the judgment module 40 is further configured to perform a weighted summation of the defect consistency score and the model uncertainty score according to a preset weight coefficient to obtain a normalized final defect judgment score.

[0185] The normalized final defect determination score is linearly transformed and mapped to a preset determination score range to obtain the target defect determination score.

[0186] This application provides a multi-branch self-learning defect detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-branch self-learning defect detection method in Embodiment 1 above.

[0187] The following is for reference. Figure 4This document illustrates a structural schematic diagram of a multi-branch self-learning defect detection device suitable for implementing embodiments of this application. The multi-branch self-learning defect detection device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The multi-branch self-learning defect detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0188] like Figure 4 As shown, the multi-branch self-learning defect detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the multi-branch self-learning defect detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multi-branch self-learning defect detection device to communicate wirelessly or wiredly with other devices to exchange data. Although a multi-branch self-learning defect detection device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0189] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0190] The multi-branch self-learning defect detection device provided in this application, employing the multi-branch self-learning defect detection method described in the above embodiments, can solve the technical problems of inaccurate defect detection and poor robustness in complex dynamic environments. Compared with the prior art, the beneficial effects of the multi-branch self-learning defect detection device provided in this application are the same as those of the multi-branch self-learning defect detection method described in the above embodiments, and other technical features of this multi-branch self-learning defect detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0191] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0192] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0193] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the multi-branch self-learning defect detection method in the above embodiments.

[0194] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0195] The aforementioned computer-readable storage medium may be included in the multi-branch self-learning defect detection device; or it may exist independently and not be assembled into the multi-branch self-learning defect detection device.

[0196] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the multi-branch self-learning defect detection device, the device performs the following actions: acquires multiple frames of images during continuous workpiece movement; performs motion compensation and illumination correction on the multiple frames to obtain aligned multiple frames; constructs a temporal consistency tensor for the neighborhood of target pixels in the aligned multiple frames; outputs a defect consistency score and a model uncertainty score for the neighborhood of target pixels through a dual-branch network based on the temporal consistency tensor; performs a weighted summation of the defect consistency score and the model uncertainty score according to preset weight coefficients to obtain a final defect determination score; and compares the final defect determination score with a dynamic defect determination threshold to determine whether the neighborhood of target pixels is a defect region.

[0197] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0199] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0200] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multi-branch self-learning defect detection method. This solves the technical problems of inaccurate defect detection and poor robustness in complex dynamic environments. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-branch self-learning defect detection method provided in the above embodiments, and will not be repeated here.

[0201] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-branch self-learning defect detection method described above.

[0202] The computer program product provided in this application can solve the technical problems of inaccurate defect detection and poor robustness in complex dynamic environments. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-branch self-learning defect detection method provided in the above embodiments, and will not be repeated here.

[0203] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A multi-branch self-learning defect detection method, characterized in that, The method includes: Acquire multiple frames of images during the continuous movement of the workpiece; Motion compensation and illumination correction are performed on the multi-frame images to obtain aligned multi-frame images; Construct a temporally consistent tensor for the neighborhood of the target pixels in the aligned multi-frame images; Based on the temporal consistency tensor, the defect consistency score and model uncertainty score of the target pixel neighborhood are output through a dual-branch network. The final defect determination score is obtained by weighting and summing the defect consistency score and the model uncertainty score according to preset weighting coefficients. The final defect determination score is compared with the dynamic defect determination threshold to determine whether the neighborhood of the target pixel is a defect region.

2. The method as described in claim 1, characterized in that, The step of performing motion compensation and illumination correction on the multi-frame images to obtain aligned multi-frame images includes: The multi-frame images are analyzed by a motion and illumination alignment learning network to obtain the predicted relative displacement parameters and illumination correction parameters between adjacent frames. The displacement compensation of the multi-frame images is performed using the relative displacement parameters to obtain a displacement-corrected image; The illumination is adjusted on the displacement-corrected image using the illumination correction parameters to obtain aligned multi-frame images.

3. The method as described in claim 1, characterized in that, The step of constructing a temporally consistent tensor for the target pixel neighborhood in the aligned multi-frame images includes: In the aligned multi-frame images, a fixed-size neighborhood centered on the target pixel is selected as the target pixel neighborhood; Calculate the average brightness value of the target pixel neighborhood in the aligned multi-frame images to obtain the average brightness value; The brightness variance is obtained by calculating the brightness variation of the target pixel neighborhood in the aligned multi-frame images. The brightness correlation of the target pixel neighborhood in the current frame image and the previous frame image is calculated to obtain the inter-frame covariance; The temporal consistency tensor is obtained based on the mean brightness, the variance of brightness, and the inter-frame covariance.

4. The method as described in claim 1, characterized in that, The step of outputting the defect consistency score and model uncertainty score of the target pixel neighborhood through a dual-branch network based on the temporal consistency tensor includes: Features of the temporal consistency tensor are extracted using a convolutional neural network; The features of the temporal consistency tensor are processed by the consistency branch of the dual-branch network to obtain the defect consistency score of the target pixel neighborhood. The features of the temporal consistency tensor are processed by the uncertainty branch of the dual-branch network to obtain the model uncertainty score of the target pixel neighborhood.

5. The method as described in claim 1, characterized in that, The step of comparing the final defect determination score with the dynamic defect determination threshold to determine whether the target pixel neighborhood is a defect region includes: The final defect determination score is compared with the dynamic defect determination threshold to obtain the comparison result; When the comparison result is that the final defect determination score is greater than the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a defect region; When the comparison result is that the final defect determination score is less than or equal to the dynamic defect determination threshold, the neighborhood of the target pixel is determined to be a non-defect region.

6. The method as described in claim 5, characterized in that, After the step of comparing the final defect determination score with the dynamic defect determination threshold to obtain the comparison result, the method further includes: Obtain the false alarm rate and false negative rate data of the comparison results; The false positive rate and false negative rate data are input into the reinforcement learning model to obtain the detection results; Based on the detection results, the dynamic defect determination threshold is adjusted and updated.

7. The method as described in claim 1, characterized in that, The step of weighting and summing the defect consistency score and the model uncertainty score according to preset weighting coefficients to obtain the final defect judgment score includes: The defect consistency score and the model uncertainty score are weighted and summed according to preset weighting coefficients to obtain the normalized final defect judgment score. The normalized final defect determination score is linearly transformed and mapped to a preset determination score range to obtain the target defect determination score.

8. A multi-branch self-learning defect detection device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of images during the continuous movement of the workpiece; The image processing module is used to perform motion compensation and illumination correction on the multi-frame images to obtain aligned multi-frame images; and to construct a temporal consistency tensor for the neighborhood of the target pixels in the aligned multi-frame images. The scoring module is used to output the defect consistency score and model uncertainty score of the target pixel neighborhood through a dual-branch network based on the temporal consistency tensor; and to obtain the final defect judgment score by weighted summation of the defect consistency score and the model uncertainty score according to preset weight coefficients. The judgment module is used to compare the final defect judgment score with the dynamic defect judgment threshold to determine whether the target pixel neighborhood is a defect region.

9. A multi-branch self-learning defect detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-branch self-learning defect detection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-branch self-learning defect detection method as described in any one of claims 1 to 7.

Images