Optical device defect detection method and system

Abstract

The invention discloses an optical device defect detection method and system, and belongs to the technical field of computer vision. According to the method, a basic training set is constructed by collecting images and performance parameters of a plurality of known defective optical devices; training a Transform model by using the basic training set to obtain a defect detection model; and acquiring an image and performance parameters of a to-be-detected device, inputting the image and the performance parameters into the defect detection model for detection, and obtaining an appearance defect detection result, a performance defect detection result and an appearance defect area coordinate. Through systematic data driving training and multi-modal information fusion, the overall performance of the defect detection model in optical device defect detection is effectively improved, and the accuracy and efficiency of optical device defect detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an optical device defect detection method and system. Background Art

[0002] Optical device defect detection mainly includes appearance defect detection and performance parameter defect detection. The relevant technology mainly uses power supply, spectrometer, oscilloscope and other equipment to directly apply preset bias and current to the optical device using DC power-on method, so as to test the working current and optical power of the optical device to judge the performance defects. On this basis, the appearance of the optical device is observed manually through optical equipment such as microscopes to judge the appearance defects of the optical device.

[0003] However, manual observation is highly dependent on individual experience, resulting in uneven appearance inspection results. It is not easy to observe the appearance defects of optical devices in all directions, which can easily lead to accidental human errors. It requires inspectors to stay at their posts for a long time without interruption, which is inefficient in large-scale rapid production. Summary of the invention

[0004] The main purpose of the present invention is to provide a method and system for detecting defects in optical devices, aiming to solve the technical problems in the related art of low efficiency and poor detection effect of defects in optical devices.

[0005] To achieve the above object, the present invention provides a method for detecting defects in an optical device, the method comprising the following steps:

[0006] S1, collect images and performance parameters of several known defective optical devices to build a basic training set;

[0007] S2, use the basic training set to train the Transformer model to obtain a defect detection model;

[0008] S3, collecting images and performance parameters of the device to be inspected, inputting the defect detection model for detection, and obtaining appearance defect detection results, performance defect detection results and appearance defect area coordinates.

[0009] In addition, to achieve the above-mentioned purpose, the present invention also provides an optical device defect detection system, the system comprising:

[0010] The multimodal data acquisition module is used to collect the performance parameters of the optical device to be tested and the appearance images from multiple angles.

[0011] The multimodal data processing module is used to standardize the collected performance parameters and appearance images before stitching them together.

[0012] The Transformer defect detection module is used to extract features from the splicing processing results and output defect detection results.

[0013] The active learning and optimization module is used to optimize the parameters of the Transformer model using the Bayesian optimization engine and trigger retraining of the Transformer model.

[0014] The present invention constructs a basic training set based on the appearance images and performance parameters of known defective optical devices, providing a solid learning foundation for the model. Then, training is performed based on the Transformer model, so that the defect detection model can effectively capture the appearance characteristics and performance parameter characteristics of known defective optical devices from the multimodal data of appearance images and performance parameters. Next, the appearance images and performance parameters of the device to be detected are collected, and the defect detection model is used for detection to obtain accurate appearance defect detection results, performance defect detection results, and appearance defect area coordinates. Through systematic data-driven training and multimodal information fusion, the present invention effectively improves the overall performance of the defect detection model in optical device defect detection, and significantly improves the accuracy and efficiency of optical device defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a process of an embodiment of a method for detecting defects in an optical device according to the present invention;

[0016] Figure 2 It is a structural schematic diagram of a data acquisition device in an embodiment of the optical device defect detection method of the present invention;

[0017] Figure 3 It is a structural schematic diagram of a Transformer model in an embodiment of the optical device defect detection method of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of the encoder in the Transformer model;

[0019] Figure 5 It is a structural schematic diagram of the optical device defect detection system of the present invention.

[0020] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0022] The inventive concept of the present application is further described below in conjunction with some specific embodiments and implementation methods.

[0023] The embodiment of the present invention provides a method for detecting defects in an optical device, referring to Figure 1 , Figure 1 The present invention is a flowchart of an optical device defect detection method embodiment of the present invention.

[0024] In this embodiment, the optical device defect detection method specifically includes the following steps:

[0025] Step S1: Collect images and performance parameters of several known defective optical devices to construct a basic training set.

[0026] The step S1 specifically includes the following steps:

[0027] Step S11: using a data acquisition device to acquire images of known defective optical devices at multiple angles.

[0028] like Figure 2 As shown, Figure 2 (a) is the main view. Figure 2 (b) is a top view. The data acquisition equipment includes a carrier 1, an integrated system device 2, a plurality of industrial CCD cameras 3 (for example, 6), an optical fiber 7, a liftable vacuum adsorption sticker 8, and a rotatable shuttle device 9. The carrier 8 is located in the center of the rotatable shuttle device 9, and the 6 industrial CCD cameras 3 are fixed at intervals on the edge of the rotatable shuttle device 9. The liftable vacuum adsorption sticker 8 is located in the center of the upper surface of the carrier 1. The known defective optical device includes an FPC flexible board 4, a connecting pad 5, and a device body 6 connected in sequence.

[0029] First, the known defect optical device is fixed on the upper surface of the liftable vacuum adsorption post 8, and is lifted and lowered vertically along with the liftable vacuum adsorption post 8. The integrated system device 2 is connected to the device body 6 through the optical fiber 7 to collect the performance parameters of the known defect optical device. The integrated system device 2 is connected to the device body 6 through the FPC flexible board 4 to power the known defect optical device.

[0030] When receiving the acquisition instruction, the rotatable Shuttle device 9 drives the 6 industrial CCD cameras 3 to rotate at an angular velocity of ω=15° / s, and the vacuum adsorption sticker 8 thereon drives the known defective optical device to rise at a speed of 2mm / s. At the same time, the 6 industrial CCD cameras 3 shoot the known defective optical device at 6 preset angles (0, 60, 120, 180, 240, 300), and obtain 36 images shot at 6 angles.

[0031] Step S12: collecting performance parameters of known defective optical devices through data collection equipment.

[0032] During each shooting, the integrated system device 2 collects the performance parameters of the known defective optical device through the GPIB interface, and obtains a total of multiple groups of performance parameters (for example, 6 groups, each group of performance parameters corresponds to an image at one angle), and each group of performance parameters specifically includes electrical parameters and optical parameters. Among them, the electrical parameters include bias current, voltage and impedance, and the optical parameters include optical power and spectral flatness.

[0033] Step S13: The collected images, performance parameters and actual defects are used as a basic training set.

[0034] In step S1, multiple cameras rotate around the known defective optical device in the vertical ascent, and collect images of the known defective optical device from different angles, so as to comprehensively obtain the appearance information of the known defective optical device, avoid the defect recognition bias caused by a single viewing angle, ensure that the detection results are more objective and comprehensive, and enhance the subsequent model's ability to recognize subtle defects at different viewing angles.

[0035] Step S2: Use the basic training set to train the Transformer model to obtain a defect detection model.

[0036] The step S2 specifically includes the following steps:

[0037] Step S21: performing standardization processing on the basic training set to obtain a plurality of preprocessed images and a plurality of groups of preprocessed parameter vectors.

[0038] The number of preprocessed images is the same as the number of groups of preprocessing parameter vectors, and each group of preprocessing parameter vectors corresponds to one preprocessed image.

[0039] The step S21 specifically includes the following steps:

[0040] Step S21 - 1 : For each image in the basic training set, a resizing process is first performed, a color correction process is then performed, and finally a normalization process is performed to obtain the preprocessed image.

[0041] Size scaling: The size of the captured image is scaled to 512×512 through bicubic difference scaling. Specifically, first, the scaling ratio is calculated based on the captured image size and the target size; each pixel of the target image corresponding to the target size is mapped back to the captured image according to the scaling ratio to obtain the position coordinates of each pixel in the captured image; 16 neighboring pixels in a 4×4 area around each position coordinate are selected from the captured image, and the weight of each neighboring pixel is assigned according to the distance using the bicubic kernel function, and then weighted summation is performed based on the weights, and the weighted summation result is used as the target pixel of the corresponding position coordinate of the target image. When the boundary exceeds the limit, mirror filling or edge extension processing is used to finally obtain the target image scaled to 512×512.

[0042] Color correction processing: Retinex algorithm is used to eliminate the reflection interference of the target image:

[0043] ;

[0044] in, Indicates that the target image is at coordinates The original pixel value at ; Represents Gaussian filtering, which is used to smooth the target image and eliminate reflection interference; represents a joint convolution operation, which is used to apply a Gaussian filter to the target image; Represents the pixel values ​​of the image after color correction.

[0045] Image normalization: Calculate the mean of each channel for the image processed by the Retinex algorithm. , Standard Deviation , and then perform Z-score processing to obtain the preprocessed image.

[0046] Step S21 - 2 : After normalizing each group of performance parameters in the basic training set, outlier filtering is performed to obtain the preprocessing parameter vector.

[0047] Parameter normalization: Convert the performance parameters into a parameter vector X of dimension d=8:

[0048] ;

[0049] in, Indicates the DC bias current when the optical device is working; Indicates the power supply voltage of the optical device; Indicates the impedance of the optical device; Indicates optical power; Indicates spectral flatness; Indicates the current fluctuation rate; Indicates optical power deviation; Indicates the shift in the center wavelength of the spectrum.

[0050] Normalize the parameter vector:

[0051] ;

[0052] in, Represents the i-th parameter in the parameter vector X; Representation parameters The normalized value, represents the mean of the i-th parameter, Represents the standard deviation of the ith parameter.

[0053] Outlier filtering: If , then determine the performance parameters If the collection is abnormal, re-collect and finally obtain the preprocessing parameter vector .

[0054] In step S21, the image and performance parameters are preprocessed and normalized respectively, which enhances the quality and consistency of the training data, helps to improve the stability, convergence speed and final detection accuracy during the model training process, reduces the impact of noise and inconsistency, and thus improves the performance of optical device defect detection.

[0055] Step S22: performing feature encoding on each of the preprocessed images to obtain a corresponding image feature vector; performing feature encoding on each group of the preprocessed parameter vectors to obtain a corresponding parameter feature vector.

[0056] Among them, the image feature vector corresponds to the parameter feature vector one by one.

[0057] The step S22 specifically includes the following steps:

[0058] Step S22-1: Use the improved ResNet-50 model to perform feature extraction processing on each preprocessed image to obtain a first feature map.

[0059] Among them, the improved ResNet-50 model is a conventional ResNet-50 model with the last two fully connected layers removed.

[0060] Step S22-2: Use the learnable position encoding matrix to perform position encoding processing on the first feature map to obtain an image feature vector :

[0061] ;

[0062] in, Represents the first feature map output by the improved ResNet-50 model, with a dimension of 16×16×2048; Indicates the flattening of the feature map, the first feature map Flattened to a 256×2048 feature sequence; represents a learnable position encoding matrix, which is used to add spatial position information to the feature sequence; The dimension is 256×2048; Represents a dimension of 256×2048.

[0063] Step S22-3: Preprocess each set of parameter vectors through two fully connected layers Processing is performed to obtain the parameter feature vector:

[0064] ;

[0065] Among them, the preprocessing parameter vector The dimension is 1×8; Represents the weight matrix of the first fully connected layer, which is used to map the preprocessing parameter vector to 512 dimensions; represents the bias term of the first fully connected layer; Represents the weight matrix of the second fully connected layer, which is used to map the output of the first fully connected layer to 2048 dimensions; represents the bias term of the second fully connected layer; Represents the Gaussian error linear unit activation function, which is used to enhance nonlinear expression capabilities; Represents the encoded parameter feature vector, with a dimension of 1×2048; Indicates the dimension is 1×2048.

[0066] In the entire step S22, the improved ResNet-50 model extracts deep features, effectively capturing complex patterns and tiny defects in the image; the learnable position encoding retains the spatial position information of the image, enhancing the model's sensitivity to defect locations and spatial structures; the two fully connected layers integrate performance parameter vectors to improve the understanding of the complex relationship between parameters, further optimizing the accuracy and comprehensive analysis capabilities of defect detection. Overall, the synergistic effect improves the accuracy, robustness, and detail capture capabilities of the model in defect detection.

[0067] Step S23: After the image feature vector and the corresponding parameter feature vector are concatenated, they are input into the Transformer model for processing to obtain the appearance defect prediction result, the appearance defect area coordinates and the performance defect prediction result.

[0068] Among them, Figure 3 As shown, the Transformer model includes a six-layer encoder and a detection unit with the same structure, and the step S23 specifically includes the following steps:

[0069] Step S23-1: Concatenate the image feature vector and the corresponding parameter feature vector to obtain the concatenated first feature vector :

[0070] ;

[0071] in, Indicates concatenation processing; the first eigenvector It is also the original input of the first layer encoder.

[0072] Step S23-2: Use a six-layer encoder to transfer layer by layer, perform feature extraction processing on the first feature vector, and obtain image branch features and performance parameter branch features.

[0073] Among them, Figure 4 As shown, each layer of the encoder includes a multi-head self-attention processing layer, a layer normalization processing layer 1, a feedforward network processing layer, and a layer normalization processing layer 2 connected in sequence, each multi-head self-attention processing layer includes eight self-attention heads, and the single-head dimension is 257 dimensions. The step S23-2 specifically includes the following steps:

[0074] Step S23-2-1: Input the first feature vector into the eight self-attention heads of the first layer encoder for processing, thereby obtaining eight self-attention processing results:

[0075] ;

[0076] in, Represents the self-attention processing result of the i-th self-attention head; represents the query matrix of the i-th self-attention head; represents the query weight matrix of the i-th self-attention head; represents the key matrix of the i-th self-attention head; represents the key weight matrix of the i-th self-attention head; Represents the value matrix of the i-th self-attention head; Represents the value weight matrix of the i-th self-attention head; Represents scaled dot-product attention processing.

[0077] Step S23-2-2: After concatenating the eight self-attention processing results, map them back to the input dimension to obtain the multi-head self-attention processing result :

[0078] ;

[0079] in, represents the concatenation process, which concatenates the eight self-attention processing results into a 2056×2048 matrix; Represents the output projection matrix, which is used to map the concatenated matrix back to the input dimension (257×2048 dimensions).

[0080] Step S23-2-3: After processing the multi-head self-attention processing results using the layer normalization processing layer 1, the dimension is expanded to four times using the feedforward network processing layer to obtain the processing results of the feedforward network processing layer in the first layer encoder :

[0081] ;

[0082] in, Represents the first layer fully connected weight matrix, with a dimension of 2048×8192, which is used to expand the dimension of the first eigenvector to 8192 dimensions; Represents the second-layer fully connected weight matrix, with a dimension of 8192×2048, which is used to compress the output of the first-layer fully connected back to 2048 dimensions; Represents the bias term of the first fully connected layer; Represents the bias term of the second fully connected layer; Represents the activation function, which is used to introduce nonlinear transformation; Representation layer normalization processing.

[0083] Step S23-2-4: Use the layer normalization processing layer 2 to process the processing result of the feedforward network processing layer in the first layer encoder to obtain the output result of the first layer encoder :

[0084] ;

[0085] Step 23-2-5: The output result of the first layer encoder is passed through the remaining five layers of encoders layer by layer to obtain the second feature vector :

[0086] ;

[0087] in, represents feed-forward network processing; indicates multi-headed attention processing; Represents the output of the encoder at the fifth layer.

[0088] Step S23-2-6: For the second eigenvector Decompose and obtain image branch features and performance parameter branch characteristics .

[0089] In the entire step S23-2, the image feature vector and the parameter feature vector are spliced ​​and input into the Transformer model for training at the same time, which can extract features of the appearance data and performance parameter data of known defective optical devices at the same time, thereby improving the training efficiency of the model. At the same time, the six-layer encoder is used to process the spliced ​​first feature vector layer by layer, which can gradually optimize the feature information and obtain a higher-level feature representation. Overall, the model's ability to process complex data is enhanced, thereby more accurately identifying and analyzing optical device defects.

[0090] Step S23-3: Using the detection unit to detect image branch features and performance parameter branch characteristics Process to obtain the appearance defect prediction result, the coordinates of the appearance defect area, and the performance defect prediction result.

[0091] The step S23-3 specifically includes the following steps:

[0092] Step S23-3-1: Perform average pooling on the image branch features to obtain global features :

[0093] ;

[0094] Among them, represents the i-th image feature vector of the image branch features.

[0095] Step S23-3-2: Perform mapping processing on the global features through the third fully connected layer, and output the appearance defect category probability :

[0096] ;

[0097] Among them, K represents the number of appearance defect categories (such as solder joint voids, scratches, bubbles, which are not limited in this embodiment); represents the weight matrix of the third fully connected layer, which is used to map the 2048-dimensional global features to the K-dimensional category space; represents the bias term of the third fully connected layer, and each category corresponds to a bias value; represents the normalization process, which is used to convert the K-dimensional scores after linear transformation into a probability distribution to ensure that the sum of all category probabilities is 1.

[0098] Step S23-3-3: Determine the category with the highest probability as the appearance defect prediction result according to the appearance defect category probability.

[0099] Step S23-3-4: Use the Deformable DETR decoder to process the image branch features to obtain the coordinates of the appearance defect area :

[0100] ;

[0101] Among them, represents the learnable anchor query matrix, which contains N = 100 query vectors, each vector dimension is 256, and is used to locate potential defect areas; represents the image feature vector encoded by the 6th layer encoder, which contains the spatial information (such as height H, width W) and channel dimension C of the image; It represents a deformable attention mechanism, which can focus the anchor query on the defect-related regions in the image features by dynamically sampling points; It represents a multi-layer perceptron, which is used to map the image feature vector after attention processing to the coordinates of the defect bounding box (such as the center point, width, and height); It represents an activation function, which is used to normalize the coordinate values to the range of [0, 1].

[0102] Step S23-3-5: Perform mapping processing on the performance parameter branch features through the fourth fully connected layer to obtain performance indicators :

[0103] ;

[0104] Among them, It represents the performance parameter branch features, with a dimension of 1×2048, containing the high-level features encoded by the performance parameters (such as current and voltage) of the optical device; It represents the number of performance indicators (such as performance indicators such as optical power deviation and current volatility); It represents the weight matrix of the fourth fully connected layer, which is used to map the 2048-dimensional parameter feature vector to the M-dimensional performance indicator space; It represents the bias term of the fourth fully connected layer, which is used to adjust the baseline offset of the mapping result.

[0105] Step S23-3-6: Use the Softmax function to process the performance indicators to obtain the performance defect category probability, and determine the category with the highest probability as the performance defect prediction result.

[0106] In the entire step S23-3, on the basis of using two fully connected layers to perform mapping processing on the image branch features and the performance parameter branch features respectively to obtain the appearance defect prediction result and the performance defect prediction result, the Deformable DETR decoder is used to further process the image branch features to obtain the specific coordinates of the appearance defect area. On the basis of realizing defect detection, the accurate positioning of appearance defects is further realized, making the defect detection result more valuable for reference.

[0107] Step S24: Use the Bayesian optimization strategy to optimize the parameters of the Transformer model and retrain the Transformer model to obtain the defect detection model.

[0108] The specific steps of step S24 include:

[0109] Step S24-1: Through the MC-Dropout method, perform T forward propagations to obtain the average prediction value and the total uncertainty:

[0110] ;

[0111] Among them, represents the predicted value of the t-th forward propagation of the sample x; represents the predicted variance of the t-th forward propagation of the sample x; represents the number of sampling times, taking 50, which is used to estimate the model uncertainty; represents performing times of sampling on the sample x; the average predicted value; represents the total uncertainty, including the uncertainty of the model itself and data noise; the dropout rate p of MC-Dropout takes 0.1; represents the variance calculation.

[0112] Step S24-2: Construct an acquisition function based on the average predicted value to quantify the potential optimization value of the sample x:

[0113] ;

[0114] Among them, represents the expected improvement of the sample x; represents the expectation calculation; represents the calculation of the expected improvement amount; represents the exploration weight, taking 0.2, which is used to balance "exploiting the known optimum" and "exploring new regions"; represents the predicted standard deviation of the model for the sample x; represents the current optimal objective value.

[0115] Step S24-3: Based on the acquisition function, use the L-BFGS-B algorithm to solve the optimal parameters of the Transformer model:

[0116] ;

[0117] Among them, represents the next optimal parameter point found by the optimization algorithm; represents finding the parameter that makes maximize; represents the constraint condition.

[0118] Step S24-4: Obtain new samples (new input-output pairs) based on the optimal parameter point, update the basic training set, and obtain the updated training set:

[0119] ;

[0120] Among them, represents the basic training set, which contains the existing input-output pairs; represents the updated training set, which contains the basic training set and new samples; Indicates the true value obtained by actual measurement of the optimal parameter point for the actual measurement

[0121] Step S24-5: Based on the total uncertainty obtain the standard deviation threshold , if it is detected that , then retrain the model using the updated training set, thereby constructing a defect detection model

[0122] wherein represents the predicted mean of the defect detection model for the optimal parameter point ; represents the standard deviation threshold

[0123] In the whole step S24, the MC-Dropout method calculates the average predicted value and the total uncertainty through multiple forward propagations, which can help the model quantify the uncertainty and provide more reliable prediction results. The acquisition function is constructed based on the average predicted value, and the L-BFGS-B algorithm is combined to solve the optimal parameters, effectively optimizing the performance of the defect detection model. The updated training set enhances the generalization ability of the model by introducing new input-output pair samples. Finally, based on the standard deviation threshold, when samples with large prediction errors are detected, the model is retrained to further improve the detection accuracy of the model for optical device defects. Generally speaking, by introducing the uncertainty quantification and adaptive optimization strategy, the continuous improvement and self-optimization of the defect detection model are realized, so that it performs more precisely and stably in complex environments

[0124] Step S3: Collect the images and performance parameters of the device to be detected, input them into the defect detection model for detection, and obtain the appearance defect detection result, the performance defect detection result, and the coordinates of the appearance defect area

[0125] In this embodiment, a basic training set is constructed based on the appearance images and performance parameters of known defective optical devices, providing a solid learning foundation for the model. Then, training is carried out based on the Transformer model, enabling the defect detection model to effectively capture the appearance features and performance parameter features of known defective optical devices from multi-modal data of appearance images and performance parameters. Next, the appearance images and performance parameters of the device to be detected are collected and detected through the defect detection model to obtain accurate appearance defect detection results, performance defect detection results, and the coordinates of the appearance defect area. The present invention effectively improves the overall performance of the defect detection model in the detection of optical device defects through systematic data-driven training and multi-modal information fusion, significantly improving the accuracy and efficiency of optical device defect detection

[0126] As shown in Table 1, this solution comprehensively leads the traditional solution in multiple dimensions such as detection accuracy, detection efficiency, multi-modal data fusion, adaptability to complex defects, labor cost, environmental robustness, scalability, and real-time feedback optimization. Therefore, this solution has obvious advantages over the traditional solution in the defect detection of optical devices.

[0127] Table 1: Comparison of the advantages between this solution and the traditional solution:

[0128]

[0129] Furthermore, as Figure 5 shown, to achieve the above purpose, the present invention also provides an optical device defect detection system, which may include;

[0130] A multi-modal data acquisition module, configured to acquire the performance parameters and multi-angle appearance images of the optical device to be detected.

[0131] A multi-modal data processing module, configured to first perform normalization processing on the acquired performance parameters and appearance images, and then perform stitching processing.

[0132] A Transformer defect detection module, configured to extract features from the stitching processing result and output a defect detection result.

[0133] An active learning and optimization module, configured to optimize the parameters of the Transformer model and trigger the retraining of the Transformer model by using a Bayesian optimization engine.

[0134] Furthermore, the multi-modal data acquisition module may include an industrial CCD camera array unit, configured to acquire multi-angle appearance images of the device to be detected; and a performance parameter tester unit, configured to acquire performance parameters of the device to be detected.

[0135] The multi-modal data processing module may include an image preprocessing unit, configured to perform normalization processing on the acquired appearance images; a performance parameter normalization unit, configured to perform normalization processing on the acquired performance parameters; and a feature alignment and stitching unit, configured to align and stitch the output results of the image preprocessing unit and the performance parameter normalization unit.

[0136] The Transformer defect detection module may include a multi-modal fusion encoder unit, configured to perform feature extraction processing on the output result of the multi-modal data processing module; and a multi-task detection head unit, configured to perform mapping processing and positioning processing on the feature extraction result, and output an appearance defect detection result, a performance defect detection result, and the coordinates of the appearance defect area.

[0137] The active learning and optimization module may include a Bayesian optimization engine unit for optimizing the parameters of the Transformer model to solve for the optimal parameters, and a model retraining module for retraining the Transformer model based on the uncertainty of the model.

[0138] It should be noted that the functions that can be achieved by each module in the optical device defect detection system provided in this embodiment and the corresponding technical effects can be referred to the descriptions of the specific implementations in the various embodiments of the optical device defect detection method of the present invention. For the sake of brevity of the specification, they will not be elaborated here.

[0139] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0140] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformations made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, are similarly included in the patent protection scope of the present invention.

Claims

1. A method for detecting defects in optical devices, characterized in that: The method comprises the following steps: S1, collect images and performance parameters of several known defective optical devices to build a basic training set; S2, use the basic training set to train the Transformer model to obtain a defect detection model; S3, collecting images and performance parameters of the device to be inspected, inputting the defect detection model for detection, and obtaining appearance defect detection results, performance defect detection results and appearance defect area coordinates.

2. The optical device defect detection method according to claim 1, characterized in that: The S1 specifically includes: S11, collecting images of known defective optical devices at multiple angles through a data acquisition device; S12, collecting performance parameters of known defective optical devices through a data acquisition device; S13, taking the collected images, performance parameters and actual real defects as a basic training set.

3. The optical device defect detection method according to claim 1, characterized in that: The S2 specifically includes: S21, performing standardization processing on the basic training set to obtain multiple preprocessed images and multiple groups of preprocessed parameter vectors; S22: performing feature coding on each of the preprocessed images to obtain a corresponding image feature vector; performing feature coding on each group of the preprocessed parameter vectors to obtain a corresponding parameter feature vector; S23, after splicing the image feature vector and the corresponding parameter feature vector, input the concatenated vector into the Transformer model for feature extraction processing to obtain appearance defect prediction results, appearance defect region coordinates, and performance defect prediction results; S24, optimizing the parameters of the Transformer model using a Bayesian optimization strategy and retraining the Transformer model to obtain the defect detection model.

4. The optical device defect detection method according to claim 3, characterized in that: The S21 specifically includes: S21-1, for each image in the basic training set, firstly resizing the image, then performing color correction, and finally performing normalization to obtain the preprocessed image; S21-2, normalizing each group of performance parameters in the basic training set, and then performing outlier filtering to obtain the preprocessing parameter vector.

5. The optical device defect detection method according to claim 3, characterized in that: The S22 specifically includes: S22-1, using an improved ResNet-50 model to process each preprocessed image to obtain a first feature map; S22-2, using a learnable position encoding matrix to perform position encoding processing on the first feature map to obtain an image feature vector : ； in, Represents the first feature map output by the improved ResNet-50 model; Indicates the flattening of the feature map; represents a learnable position encoding matrix; Indicates the dimension is 256×2048; S22-3, each set of preprocessing parameter vectors is processed through two fully connected layers Processing is performed to obtain the parameter feature vector: ； in, represents the weight matrix of the first fully connected layer; represents the bias term of the first fully connected layer; Represents the weight matrix of the second fully connected layer; represents the bias term of the second fully connected layer; represents the Gaussian error linear unit activation function; represents the encoded parameter feature vector; Indicates the dimension is 1×2048.

6. The optical device defect detection method according to claim 3, characterized in that: The Transformer model includes a six-layer encoder and a detection unit with the same structure, and S23 specifically includes: S23-1, performing splicing processing on the image feature vector and the corresponding parameter feature vector to obtain a spliced ​​first feature vector; S23-2, using a six-layer encoder to transfer layer by layer, performing feature extraction processing on the first feature vector to obtain image branch features and performance parameter branch features; S23-3, using the detection unit to process the image branch features and the performance parameter branch features to obtain appearance defect prediction results, appearance defect area coordinates and performance defect prediction results.

7. The optical device defect detection method according to claim 6, characterized in that: Each layer of the encoder includes a multi-head self-attention processing layer, a layer normalization processing layer 1, a feedforward network processing layer and a layer normalization processing layer 2 connected in sequence, each multi-head self-attention processing layer includes eight self-attention heads, and S23-2 specifically includes: S23-2-1, inputting the first feature vector into the eight self-attention heads of the first layer encoder for processing, thereby obtaining eight self-attention processing results; S23-2-2, concatenate the eight self-attention processing results and map them back to the input dimension to obtain the multi-head self-attention processing result; S23-2-3, after processing the multi-head self-attention processing result using the layer normalization processing layer 1, the dimension is expanded to four times using the feedforward network processing layer to obtain the processing result of the feedforward network processing layer in the first layer encoder; S23-2-4, using the layer normalization processing layer 2 to process the processing result of the feedforward network processing layer in the first layer encoder to obtain the output result of the first layer encoder; S23-2-5, passing the output result of the first layer encoder through the remaining five layers of encoders layer by layer to obtain a second feature vector; S23-2-6, decompose the second eigenvector to obtain image branch features and performance parameter branch features.

8. The optical device defect detection method according to claim 6, characterized in that: The S23-3 specifically includes: S23-3-1, average pooling the image branch features to obtain global features; S23-3-2, mapping the global features through the third fully connected layer, and outputting the probability of appearance defect category; S23-3-3, according to the probability of appearance defect categories, determine the category with the highest probability as the appearance defect prediction result; S23-3-4, using the Deformable DETR decoder to process the image branch features to obtain the coordinates of the appearance defect area; S23-3-5, mapping the performance parameter branch features through the fourth fully connected layer to obtain a performance indicator; S23-3-6, using the Softmax function to process the performance indicators, obtain the probability of performance defect categories, and determine the category with the highest probability as the performance defect prediction result.

9. The optical device defect detection method according to claim 3, characterized in that: The S24 specifically includes: S24-1, through the MC-Dropout method, perform T forward propagations to obtain the average prediction value and total uncertainty: ； in, Represents the predicted value of the t-th forward propagation of sample x; Represents the prediction variance of the t-th forward propagation of sample x; Indicates the number of sampling times; It means to perform The average predicted value of the subsamples; represents the total uncertainty; represents variance calculation; S24-2, construct the acquisition function based on the average prediction value: ； in, represents the expected improvement of sample x; represents the expected calculation; It indicates the expected improvement calculation; represents the exploration weight; Represents the model's predicted standard deviation for sample x; Indicates the current optimal target value; S24-3, based on the acquisition function, use the L-BFGS-B algorithm to solve the optimal parameters of the Transformer model: ； in, Represents the next optimal parameter point found by the optimization algorithm; Indicates the search for The parameter to be maximized; Indicates the DC bias current when the optical device is working; Indicates the power supply voltage of the optical device; Indicates constraints; S24-4, obtain new samples based on the optimal parameter point, update the basic training set, and obtain the updated training set: ； in, represents the basic training set; represents the updated training set; Represents the optimal parameter point The true value obtained by actual measurement; S24-5, based on total uncertainty Get the standard deviation threshold If detected , then use the updated training set to retrain the Transformer model to build a defect detection model; in, Represents the optimal parameter point of the Transformer model The predicted mean of Indicates the standard deviation threshold.

10. An optical device defect detection system, characterized in that: The system comprises: Multimodal data acquisition module, used to collect performance parameters and multi-angle appearance images of the optical device to be tested; The multimodal data processing module is used to standardize the collected performance parameters and appearance images before splicing them; Transformer defect detection module, used to extract features from the splicing processing results and output defect detection results; The active learning and optimization module is used to optimize the parameters of the Transformer model using the Bayesian optimization engine and trigger retraining of the Transformer model.

Images