A method and system for detecting defects in optical devices

Abstract

The present invention discloses a method and system for detecting defects in optical devices, belonging to the technical field of computer vision. The present invention constructs a basic training set by collecting images and performance parameters of a number of optical devices with known defects; uses the basic training set to train a Transformer model to obtain a defect detection model; collects images and performance parameters of the device to be detected, inputs them into the defect detection model for detection, and obtains appearance defect detection results, performance defect detection results and appearance defect area coordinates. Through systematic data-driven training and multi-modal information fusion, the present invention effectively improves the overall performance of the defect detection model in the defect detection of optical devices, and significantly improves the accuracy and efficiency of the defect detection of optical devices.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a method and system for detecting defects of optical devices. Background Art

[0002] Defect detection of optical devices mainly includes appearance defect detection and performance parameter defect detection. The related technology mainly uses devices such as a power supply, a spectrometer, and an oscilloscope to directly apply a preset bias voltage and current to the optical device by means of a DC power supply method, so as to test the working current and optical power of the optical device, and to judge the performance defects. On this basis, an operator observes the appearance of the optical device through an optical device such as a microscope, so as to judge the appearance defects of the optical device.

[0003] However, by observing manually, firstly, it is extremely dependent on individual experience, resulting in uneven appearance detection effects; secondly, it is not easy to observe the appearance defects of the optical device in all directions, and it is easy to cause accidental human errors; thirdly, it requires the inspectors to stay at their posts continuously for a long time, and the efficiency is low during large-scale and rapid production. Summary of the Invention

[0004] The main purpose of the present invention is to provide a method and system for detecting defects of optical devices, aiming to solve the technical problems of low efficiency and poor detection effect in the related technology for detecting defects of optical devices.

[0005] To achieve the above purpose, the present invention provides a method for detecting defects of optical devices, and the method includes the following steps:

[0006] S1, collecting images and performance parameters of a plurality of known defective optical devices to construct a basic training set;

[0007] S2, using the basic training set to train a Transformer model to obtain a defect detection model;

[0008] S3, collecting images and performance parameters of the device to be detected, inputting the device into the defect detection model for detection, and obtaining an appearance defect detection result, a performance defect detection result, and the coordinates of the appearance defect area.

[0009] In addition, to achieve the above purpose, the present invention also provides a system for detecting defects of optical devices, and the system includes:

[0010] A multimodal data acquisition module, configured to collect performance parameters and multi-angle appearance images of the optical device to be detected.

[0011] A multimodal data processing module, configured to first perform normalization processing on the collected performance parameters and appearance images, and then perform splicing processing.

[0012] The Transformer defect detection module is used to extract features from the splicing processing results and output defect detection results.

[0013] The active learning and optimization module is used to optimize the parameters of the Transformer model using the Bayesian optimization engine and trigger retraining of the Transformer model.

[0014] The present invention constructs a basic training set based on the appearance images and performance parameters of known defective optical devices, providing a solid learning foundation for the model. Then, training is performed based on the Transformer model, so that the defect detection model can effectively capture the appearance characteristics and performance parameter characteristics of known defective optical devices from the multimodal data of appearance images and performance parameters. Next, the appearance images and performance parameters of the device to be tested are collected, and the defect detection model is used for detection to obtain accurate appearance defect detection results, performance defect detection results, and appearance defect area coordinates. Through systematic data-driven training and multimodal information fusion, the present invention effectively improves the overall performance of the defect detection model in optical device defect detection, and significantly improves the accuracy and efficiency of optical device defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of a process of an embodiment of a method for detecting defects in optical devices according to the present invention;

[0016] Figure 2 Schematic diagram of the structure of the data acquisition device in the embodiment of the optical device defect detection method of the present invention;

[0017] Figure 3 Schematic diagram of the structure of the Transformer model in an embodiment of the optical device defect detection method of the present invention;

[0018] Figure 4 This is a schematic diagram of the encoder structure in the Transformer model;

[0019] Figure 5 Schematic diagram of the structure of the optical device defect detection system of the present invention.

[0020] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] The inventive concept of the present application is further described below with reference to some specific embodiments and implementation methods.

[0023] An embodiment of the present invention provides a method for detecting defects of optical devices. Refer to Figure 1 , Figure 1 which is a schematic flowchart of an embodiment of a method for detecting defects of optical devices according to the present invention.

[0024] In this embodiment, the method for detecting defects of optical devices specifically includes the following steps:

[0025] Step S1: Collect images and performance parameters of several known defective optical devices to construct a basic training set.

[0026] The step S1 specifically includes the following steps:

[0027] Step S11: Collect images of known defective optical devices from multiple angles by a data acquisition device.

[0028] As Figure 2 shown, Figure 2 the (a) in Figure 2 is the front view, and the (b) in

[0029] is the top view. The data acquisition device includes a carrier table 1, a comprehensive system device 2, multiple industrial CCD cameras 3 (such as 6), an optical fiber 7, a liftable vacuum adsorption sticker 8, and a rotatable Shuttle device 9. The carrier table 8 is located in the center of the rotatable Shuttle device 9. The 6 industrial CCD cameras 3 are fixedly spaced at the edge of the rotatable Shuttle device 9. The liftable vacuum adsorption sticker 8 is located in the center of the upper surface of the carrier table 1. The known defective optical device includes an FPC flexible board 4, a connection pad 5, and a device body 6 connected in sequence.

[0030] First, fix the known defective optical device on the upper surface of the liftable vacuum adsorption sticker 8, and it moves up and down vertically with the liftable vacuum adsorption sticker 8. The comprehensive system device 2 is connected to the device body 6 through the optical fiber 7 for collecting the performance parameters of the known defective optical device. The comprehensive system device 2 is connected to the device body 6 through the FPC flexible board 4 for supplying power to the known defective optical device.

[0031] When receiving the acquisition instruction, the rotatable Shuttle device 9 drives the 6 industrial CCD cameras 3 to rotate at an angular velocity ω = 15° / s, and the vacuum adsorption sticker 8 thereon drives the known defective optical device to rise at a speed of 2 mm / s. At the same time, the 6 industrial CCD cameras 3 take pictures of the known defective optical device at 6 preset angles (0, 60, 120, 180, 240, 300) to obtain a total of 36 captured images at 6 angles.

[0032] Each time a shot is taken, the integrated system device 2 collects the performance parameters of the known defective optical device through the GPIB interface, obtaining multiple sets of performance parameters in total (for example, 6 sets, and each set of performance parameters corresponds to an image at a certain angle). Each set of performance parameters specifically includes electrical parameters and optical parameters. Among them, the electrical parameters include bias current, voltage, and impedance, and the optical parameters include optical power and spectral flatness.

[0033] Step S13: Use the collected images, performance parameters, and actual real defects as the basic training set.

[0034] In step S1, multiple cameras rotate around the vertically rising known defective optical device to collect images of the known defective optical device from different angles, which can comprehensively obtain the appearance information of the known defective optical device, avoid the defect recognition deviation caused by a single perspective, ensure that the detection results are more objective and comprehensive, and enhance the recognition ability of the subsequent model for subtle defects under different perspectives.

[0035] Step S2: Use the basic training set to train the Transformer model to obtain a defect detection model.

[0036] The specific steps of step S2 include the following steps:

[0037] Step S21: Perform normalization processing on the basic training set to obtain multiple preprocessed images and multiple sets of preprocessing parameter vectors.

[0038] Among them, the number of preprocessed images is the same as the number of sets of preprocessing parameter vectors, and each set of preprocessing parameter vectors corresponds to one preprocessed image.

[0039] The specific steps of step S21 include the following steps:

[0040] Step S21-1: For each image in the basic training set, first perform size scaling processing, then perform color correction processing, and finally perform normalization processing to obtain the preprocessed image.

[0041] Size scaling processing: Scale the size of the captured image to 512×512 through bicubic interpolation. Specifically, first calculate the scaling ratio according to the size of the captured image and the target size; map each pixel of the target image corresponding to the target size back to the captured image according to the scaling ratio to obtain the position coordinates of each pixel in the captured image; select 16 adjacent pixels in the 4×4 area around each position coordinate from the captured image, use the bicubic kernel function to distribute the weights of each adjacent pixel according to the distance, and then perform weighted summation according to the weights. Take the weighted summation result as the target pixel at the corresponding position coordinate of the target image. When the boundary is exceeded, use mirror filling or edge extension processing to finally obtain the target image scaled to 512×512.

[0042] Color correction process: The Retinex algorithm is used to eliminate the reflection interference of the target image:

[0043] ;

[0044] Among them, represents the original pixel value of the target image at the coordinate ; represents Gaussian filtering processing, which is used to smooth the target image and eliminate reflection interference; represents the joint convolution operation, which is used to apply the Gaussian filter to the target image; represents the pixel value of the image after color correction.

[0045] Image normalization process: For the image processed by the Retinex algorithm, calculate the mean value , standard deviation for each channel, and then perform Z-score processing to obtain the preprocessed image.

[0046] Step S21-2: After normalizing each set of performance parameters in the basic training set, perform outlier filtering to obtain the preprocessed parameter vector.

[0047] Parameter normalization process: Convert the performance parameters into a parameter vector X with dimension d = 8:

[0048] ;

[0049] Among them, represents the DC bias current when the optical device is working; represents the supply voltage of the optical device; represents the impedance of the optical device; represents the optical power; represents the spectral flatness; represents the current volatility; represents the optical power deviation; represents the spectral center wavelength shift.

[0050] Normalize the parameter vector:

[0051] ;

[0052] Among them, represents the i-th parameter in the parameter vector X; represents the value after normalizing the parameter , represents the mean value of the i-th parameter, represents the standard deviation of the i-th parameter.

[0053] Outlier filtering process: If , it is determined that the performance parameter is abnormally collected, and the collection is restarted, and finally a preprocessing parameter vector is obtained .

[0054] In step S21, the image and the performance parameter are preprocessed and normalized respectively, which enhances the quality and consistency of the training data, helps to improve the stability, convergence speed and final detection accuracy in the model training process, reduces the influence of noise and inconsistency, and thus improves the performance of the optical device defect detection.

[0055] Step S22: Perform feature encoding on each of the preprocessed images to obtain a corresponding image feature vector; perform feature encoding on each group of the preprocessed parameter vectors to obtain a corresponding parameter feature vector.

[0056] Among them, the image feature vector and the parameter feature vector correspond one by one.

[0057] The step S22 specifically includes the following steps:

[0058] Step S22-1: Use an improved ResNet-50 model to perform feature extraction processing on each preprocessed image to obtain a first feature map.

[0059] Among them, the improved ResNet-50 model is a model obtained by removing the last two fully connected layers of the conventional ResNet-50 model.

[0060] Step S22-2: Use a learnable position encoding matrix to perform position encoding processing on the first feature map to obtain an image feature vector :

[0061] ;

[0062] Among them, represents the first feature map output by the improved ResNet-50 model, with a dimension of 16×16×2048; represents the flattening process of the feature map, and flattens the first feature map into a feature sequence of 256×2048; represents a learnable position encoding matrix, which is used to add spatial position information to the feature sequence; has a dimension of 256×2048; represents a dimension of 256×2048.

[0063] Step S22-3: Process each group of preprocessed parameter vectors through two fully connected layers to obtain a parameter feature vector:

[0064] ;

[0065] Among them, the preprocessing parameter vector has a dimension of 1×8; represents the weight matrix of the first fully connected layer, which is used to map the preprocessing parameter vector to 512 dimensions; represents the bias term of the first fully connected layer; represents the weight matrix of the second fully connected layer, which is used to map the output of the first fully connected layer to 2048 dimensions; represents the bias term of the second fully connected layer; represents the Gaussian error linear unit activation function, which is used to enhance the non-linear expression ability; represents the encoded parameter feature vector, with a dimension of 1×2048; represents a dimension of 1×2048.

[0066] In the entire step S22, deep features are extracted through an improved ResNet-50 model, effectively capturing complex patterns and minute defects in the image; learnable position encoding retains the spatial position information of the image, enhancing the model's sensitivity to the defect position and spatial structure; the two fully connected layers then improve the understanding of the complex relationships between parameters by integrating the performance parameter vector, further optimizing the accuracy and comprehensive analysis ability of defect detection. Overall, they work synergistically to improve the accuracy, robustness, and detail capture ability of the model in defect detection.

[0067] Step S23: After splicing the image feature vector and the corresponding parameter feature vector, input them into the Transformer model for processing to obtain the appearance defect prediction result, the coordinates of the appearance defect area, and the performance defect prediction result.

[0068] Among them, as Figure 3 shown, the Transformer model includes six layers of encoders and detection units with the same structure. The specific steps of step S23 are as follows:

[0069] Step S23-1: Perform splicing processing on the image feature vector and the corresponding parameter feature vector to obtain the first spliced feature vector :

[0070] ;

[0071] Among them, represents the splicing processing; the first feature vector is also the original input of the first layer of encoder.

[0072] Step S23-2: Use the six layers of encoders to pass through layer by layer to perform feature extraction processing on the first feature vector to obtain the image branch feature and the performance parameter branch feature.

[0073] Among them, as Figure 4 shown, each layer of the encoder includes a multi-head self-attention processing layer, a layer normalization processing layer 1, a feed-forward network processing layer, and a layer normalization processing layer 2 that are connected in sequence. Each multi-head self-attention processing layer includes eight self-attention heads, and the single-head dimension is 257 dimensions. The specific steps of step S23-2 are as follows:

[0074] Step S23-2-1: Input the first feature vector into the eight self-attention heads of the first layer of the encoder for processing, so as to obtain eight self-attention processing results:

[0075] ; [[ID=1Z]]

[0076] Among them, represents the self-attention processing result of the i-th self-attention head; represents the query matrix of the i-th self-attention head; represents the query weight matrix of the i-th self-attention head; represents the key matrix of the i-th self-attention head; represents the key weight matrix of the i-th self-attention head; represents the value matrix of the i-th self-attention head; represents the value weight matrix of the i-th self-attention head; represents the scaled dot-product attention mechanism processing.

[0077] Step S23-2-2: After splicing the eight self-attention processing results, map them back to the input dimension to obtain the multi-head self-attention processing result :

[0078] ;

[0079] Among them, represents the splicing process, splicing the eight self-attention processing results into a 2056×2048 matrix; represents the output projection matrix, which is used to map the matrix after the splicing process back to the input dimension (257×2048 dimensions).

[0080] Step S23-2-3: After using the layer normalization processing layer 1 to process the multi-head self-attention processing result, use the feed-forward network processing layer to expand the dimension to four times to obtain the processing result of the feed-forward network processing layer in the first layer of the encoder :

[0081] ;

[0082] Among them, Represents the first layer fully connected weight matrix, with a dimension of 2048×8192, which is used to expand the dimension of the first eigenvector to 8192 dimensions; Represents the second-layer fully connected weight matrix, with a dimension of 8192×2048, which is used to compress the output of the first-layer fully connected layer back to 2048 dimensions; Represents the bias term of the first fully connected layer; Represents the bias term of the second fully connected layer; Represents the activation function, which is used to introduce nonlinear transformation; Representation layer normalization processing.

[0083] Step S23-2-4: Use layer normalization processing layer 2 to process the processing results of the feedforward network processing layer in the first layer encoder to obtain the output results of the first layer encoder :

[0084] ;

[0085] Step 23-2-5: The output of the first layer encoder is passed through the remaining five layers of encoders layer by layer to obtain the second feature vector :

[0086] ;

[0087] in, represents feedforward network processing; Indicates multi-headed attention processing; Represents the output of the encoder at the fifth layer.

[0088] Step S23-2-6: For the second eigenvector Decompose and obtain image branch features and performance parameter branch characteristics .

[0089] In step S23-2, the image feature vector and parameter feature vector are concatenated and fed into the Transformer model for training. This allows for simultaneous feature extraction of both the appearance data and performance parameter data of known defective optical devices, improving model training efficiency. Simultaneously, a six-layer encoder processes the concatenated first feature vector layer by layer, gradually optimizing feature information and obtaining higher-level feature representations. Overall, this enhances the model's ability to process complex data, enabling more accurate identification and analysis of optical device defects.

[0090] Step S23-3: Use the detection unit to detect image branch features and performance parameter branch characteristics Process to obtain the appearance defect prediction result, the coordinates of the appearance defect area, and the performance defect prediction result.

[0091] Step S23-3 specifically includes the following steps:

[0092] Step S23-3-1: Perform average pooling on the image branch features to obtain global features :

[0093] ;

[0094] Among them, represents the i-th image feature vector of the image branch features.

[0095] Step S23-3-2: Perform mapping processing on the global features through the third fully connected layer, and output the appearance defect category probability :

[0096] ;

[0097] Among them, K represents the number of appearance defect categories (such as false soldering, scratching, air bubbles, which are not limited in this embodiment); represents the weight matrix of the third fully connected layer, which is used to map the 2048-dimensional global features to the K-dimensional category space; represents the bias term of the third fully connected layer, and each category corresponds to a bias value; s represents the normalization process, which is used to convert the K-dimensional scores after linear transformation into a probability distribution, ensuring that the sum of all category probabilities is 1.

[0098] Step S23-3-3: According to the appearance defect category probability, determine the category with the highest probability as the appearance defect prediction result.

[0099] Step S23-3-4: Use the Deformable DETR decoder to process the image branch features to obtain the coordinates of the appearance defect area :

[0100] ;

[0101] Among them, represents the learnable anchor query matrix, which contains N = 100 query vectors, each vector dimension is 256, and is used to locate potential defect areas; represents the image feature vector encoded by the 6th layer encoder, which contains the spatial information (such as height H, width W) and channel dimension C of the image; It represents a deformable attention mechanism, which can make the anchor query focus on the defect-related regions in the image features by dynamically sampling points; It represents a multi-layer perceptron, which is used to map the image feature vector after attention processing into the coordinates of the defect bounding box (such as the center point, width, and height); It represents an activation function, which is used to normalize the coordinate values to the range of [0, 1].

[0102] Step S23-3-5: Perform mapping processing on the performance parameter branch features through the fourth fully connected layer to obtain performance metrics :

[0103] ;

[0104] Among them, It represents the performance parameter branch features, with a dimension of 1×2048, containing the high-level features encoded with the performance parameters of the optical device (such as current and voltage); It represents the number of performance metrics (such as performance metrics like optical power deviation and current volatility); It represents the weight matrix of the fourth fully connected layer, which is used to map the 2048-dimensional parameter feature vector to the M-dimensional performance metric space; It represents the bias term of the fourth fully connected layer, which is used to adjust the baseline offset of the mapping result.

[0105] Step S23-3-6: Use the Softmax function to process the performance metrics to obtain the performance defect category probabilities, and determine the category with the maximum probability as the performance defect prediction result.

[0106] In the entire step S23-3, on the basis of using two fully connected layers to perform mapping processing on the image branch features and the performance parameter branch features respectively to obtain the appearance defect prediction result and the performance defect prediction result, the Deformable DETR decoder is used to further process the image branch features to obtain the specific coordinates of the appearance defect region. On the basis of realizing defect detection, the accurate positioning of appearance defects is further realized, making the defect detection result more valuable for reference.

[0107] Step S24: Use the Bayesian optimization strategy to optimize the parameters of the Transformer model and retrain the Transformer model to obtain the defect detection model.

[0108] The specific steps of step S24 include:

[0109] Step S24-1: Through the MC-Dropout method, perform T forward propagations to obtain the average prediction value and the total uncertainty:

[0110] ;

[0111] Among them, represents the predicted value of the t-th forward propagation for sample x; represents the predicted variance of the t-th forward propagation for sample x; represents the number of sampling times, taking 50, which is used to estimate the model uncertainty; represents performing times of sampling on sample x; the average predicted value; represents the total uncertainty, including the uncertainty of the model itself and data noise; the dropout rate p of MC-Dropout takes 0.1; represents the variance calculation.

[0112] Step S24-2: Construct an acquisition function based on the average predicted value to quantify the potential optimization value of sample x:

[0113] ;

[0114] Among them, represents the expected improvement of sample x; represents the expected calculation; represents the calculation of the expected improvement amount; represents the exploration weight, taking 0.2, which is used to balance "exploiting the known optimum" and "exploring new regions"; represents the predicted standard deviation of the model for sample x; represents the current optimal objective value.

[0115] Step S24-3: Based on the acquisition function, use the L-BFGS-B algorithm to solve the optimal parameters of the Transformer model:

[0116] ;

[0117] Among them, represents the next optimal parameter point found through the optimization algorithm; represents finding the parameters that maximize ; represents the constraint condition.

[0118] Step S24-4: Obtain new samples (new input-output pairs) based on the optimal parameter point, update the basic training set, and obtain the updated training set:

[0119] ;

[0120] Among them, represents the basic training set, which contains the existing input-output pairs; represents the updated training set, which contains the basic training set and new samples; Denote the optimal parameter point The true value obtained by actual measurement

[0121] Step S24-5: Based on the total uncertainty Obtain the standard deviation threshold , if detected , then retrain the model using the updated training set, thereby constructing a defect detection model

[0122] Wherein Denote the predicted mean value of the defect detection model for the optimal parameter point ; Denote the standard deviation threshold

[0123] In the entire step S24, the MC-Dropout method calculates the average predicted value and the total uncertainty through multiple forward propagations, which can help the model quantify the uncertainty and provide more reliable prediction results. Constructing an acquisition function based on the average predicted value and solving for the optimal parameters in combination with the L-BFGS-B algorithm effectively optimizes the performance of the defect detection model. The updated training set enhances the generalization ability of the model by introducing new input-output pair samples. Finally, based on the standard deviation threshold, when samples with large prediction errors are detected, the model is retrained to further improve the detection accuracy of the model for optical device defects. Overall, by introducing uncertainty quantification and adaptive optimization strategies, continuous improvement and self-optimization of the defect detection model are achieved, so that it performs more precisely and stably in complex environments

[0124] Step S3: Acquire the image and performance parameters of the device to be detected, input them into the defect detection model for detection, and obtain the appearance defect detection result, performance defect detection result, and the coordinates of the appearance defect area

[0125] In this embodiment, a basic training set is constructed based on the appearance images and performance parameters of known defective optical devices, providing a solid learning foundation for the model. Then, training is performed based on the Transformer model, enabling the defect detection model to effectively capture the appearance features and performance parameter features of known defective optical devices from multi-modal data of appearance images and performance parameters. Next, the appearance images and performance parameters of the device to be detected are acquired and detected through the defect detection model to obtain accurate appearance defect detection results, performance defect detection results, and the coordinates of the appearance defect area. The present invention effectively improves the overall performance of the defect detection model in optical device defect detection through systematic data-driven training and multi-modal information fusion, significantly enhancing the accuracy and efficiency of optical device defect detection

[0126] As shown in Table 1, this solution comprehensively leads traditional solutions in multiple dimensions such as detection accuracy, detection efficiency, multi-modal data fusion, adaptability to complex defects, labor costs, environmental robustness, scalability, and real-time feedback optimization. Therefore, this solution has obvious advantages over traditional solutions in the detection of optical device defects.

[0127] Table 1: Comparison of the advantages of this solution and traditional solutions:

[0128]

[0129] Furthermore, as Figure 5 shown, to achieve the above objectives, the present invention also provides an optical device defect detection system, which may include;

[0130] A multi-modal data acquisition module for acquiring the performance parameters and multi-angle appearance images of the optical device to be detected.

[0131] A multi-modal data processing module for first performing standardization processing on the acquired performance parameters and appearance images, and then performing stitching processing.

[0132] A Transformer defect detection module for extracting features from the stitching processing result and outputting a defect detection result.

[0133] An active learning and optimization module for optimizing the parameters of the Transformer model using a Bayesian optimization engine and triggering retraining of the Transformer model.

[0134] Furthermore, the multi-modal data acquisition module may include an industrial CCD camera array unit for acquiring multi-angle appearance images of the device to be detected; a performance parameter tester unit for acquiring performance parameters of the device to be detected.

[0135] The multi-modal data processing module may include an image preprocessing unit for performing standardization processing on the acquired appearance images; a performance parameter standardization unit for performing standardization processing on the acquired performance parameters; a feature alignment and stitching unit for aligning and stitching the output results of the image preprocessing unit and the performance parameter standardization unit.

[0136] The Transformer defect detection module may include a multi-modal fusion encoder unit for performing feature extraction processing on the output result of the multi-modal data processing module; a multi-task detection head unit for performing mapping processing and positioning processing on the feature extraction result, and outputting appearance defect detection results and performance defect detection results along with the coordinates of the appearance defect area.

[0137] The active learning and optimization module may include a Bayesian optimization engine unit for optimizing the parameters of the Transformer model to solve for the optimal parameters, and a model retraining module for retraining the Transformer model according to the uncertainty of the model.

[0138] It should be noted that the functions that can be realized by each module in the optical device defect detection system provided in this embodiment and the corresponding technical effects achieved can refer to the descriptions of the specific implementation manners in each embodiment of the optical device defect detection method of the present invention. For the sake of brevity of the specification, they will not be elaborated here.

[0139] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0140] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformations made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, are equally included in the patent protection scope of the present invention.

Claims

1. A method for detecting defects of an optical device, characterized in that The method includes the following steps: S1. Collect images and performance parameters of several known defective optical devices to construct a basic training set; the performance parameters include optical power, spectral flatness, current, voltage, and impedance; S2. Use the basic training set to train a Transformer model to obtain a defect detection model; Specifically, S2 includes: S21. Perform normalization processing on the basic training set to obtain multiple preprocessed images and multiple groups of preprocessed parameter vectors; S22: Perform feature encoding on each preprocessed image to obtain a corresponding image feature vector; perform feature encoding on each group of preprocessed parameter vectors to obtain a corresponding parameter feature vector; S23. After splicing the image feature vector and the corresponding parameter feature vector, input them into the Transformer model for feature extraction processing to obtain an appearance defect prediction result, appearance defect area coordinates, and a performance defect prediction result; S24. Use the Bayesian optimization strategy to optimize the parameters of the Transformer model and retrain the Transformer model to obtain the defect detection model; S3. Collect images and performance parameters of the device to be detected, input them into the defect detection model for detection, and obtain an appearance defect detection result, a performance defect detection result, and appearance defect area coordinates.

2. The optical device defect detection method according to claim 1, wherein Specifically, S1 includes: S11. Collect images of known defective optical devices from multiple angles through a data acquisition device; S12. Collect performance parameters of known defective optical devices through a data acquisition device; S13. Use the collected images, performance parameters, and actual real defects as the basic training set.

3. The optical device defect detection method according to claim 1, characterized in that Specifically, S21 includes: S21-1. For each image in the basic training set, first perform size scaling processing, then perform color correction processing, and finally perform normalization processing to obtain the preprocessed image; S21-2. After performing normalization processing on each group of performance parameters in the basic training set, perform outlier filtering processing to obtain the preprocessed parameter vectors.

4. The optical device defect detection method according to claim 1, wherein Specifically, S22 includes: S22-1. Use an improved ResNet-50 model to process each preprocessed image to obtain a first feature map; S22-2, perform position encoding processing on the first feature map using a learnable position encoding matrix to obtain an image feature vector : ； Among them, represents the first feature map output by the improved ResNet-50 model; represents the flattening process of the feature map; represents the learnable position encoding matrix; represents a dimension of 256×2048; S22-3, process each group of preprocessed parameter vectors through two fully connected layers to obtain parameter feature vectors: ； Among them, represents the weight matrix of the first fully connected layer; represents the bias term of the first fully connected layer; represents the weight matrix of the second fully connected layer; represents the bias term of the second fully connected layer; represents the Gaussian error linear unit activation function; represents the encoded parameter feature vector; represents a dimension of 1×2048.

5. The optical device defect detection method according to claim 1, wherein, The Transformer model includes six layers of encoders with the same structure and a detection unit. Specifically, S23 includes: S23-1. Perform splicing processing on the image feature vector and the corresponding parameter feature vector to obtain a spliced first feature vector; S23-2. Use the six layers of encoders to transmit layer by layer to perform feature extraction processing on the first feature vector to obtain an image branch feature and a performance parameter branch feature; S23-3. Use the detection unit to process the image branch feature and the performance parameter branch feature to obtain an appearance defect prediction result, appearance defect area coordinates, and a performance defect prediction result.

6. The optical device defect detection method according to claim 5, characterized in that, Each layer of the encoder includes a multi-head self-attention processing layer, a layer normalization processing layer 1, a feed-forward network processing layer, and a layer normalization processing layer 2 connected in sequence. Each multi-head self-attention processing layer includes eight self-attention heads. Specifically, S23-2 includes: S23-2-1. Input the first eigenvector into the eight self-attention heads of the first-layer encoder for processing, so as to obtain eight self-attention processing results; S23-2-2. After concatenating the eight self-attention processing results, map them back to the input dimension to obtain the multi-head self-attention processing result; S23-2-3. After using the layer normalization processing layer 1 to process the multi-head self-attention processing result, use the feed-forward network processing layer to expand the dimension to four times to obtain the processing result of the feed-forward network processing layer in the first-layer encoder; S23-2-4. Use the layer normalization processing layer 2 to process the processing result of the feed-forward network processing layer in the first-layer encoder to obtain the output result of the first-layer encoder; S23-2-5. Pass the output result of the first-layer encoder through the remaining five layers of encoders for layer-by-layer transfer processing to obtain the second eigenvector; S23-2-6. Decompose the second eigenvector to obtain the image branch feature and the performance parameter branch feature.

7. The optical device defect detection method according to claim 5, wherein The specific steps of S23-3 are as follows: S23-3-1. Perform average pooling on the image branch feature to obtain the global feature; S23-3-2. Perform mapping processing on the global feature through the third fully connected layer and output the appearance defect category probability; S23-3-3. According to the appearance defect category probability, determine the category with the highest probability as the appearance defect prediction result; S23-3-4. Use the Deformable DETR decoder to process the image branch feature to obtain the appearance defect area coordinates; S23-3-5. Perform mapping processing on the performance parameter branch feature through the fourth fully connected layer to obtain the performance index; S23-3-6. Use the Softmax function to process the performance index to obtain the performance defect category probability, and determine the category with the highest probability as the performance defect prediction result.

8. The optical device defect detection method according to claim 1, wherein The specific steps of S24 are as follows: S24-1. Through the MC-Dropout method, perform T times of forward propagation to obtain the average prediction value and the total uncertainty: ； Among them, represents the predicted value of the t-th forward propagation of the sample x; represents the predicted variance of the t-th forward propagation of the sample x; represents the number of sampling times; represents performing on the sample x times of sampling of the average predicted value; represents the total uncertainty; represents the variance calculation; S24-2. Construct an acquisition function based on the average prediction value: ； Among them, represents the expected improvement of sample x; represents the expected calculation; represents the calculation of the expected improvement amount; represents the exploration weight; represents the predicted standard deviation of the model for sample x; represents the current optimal objective value; S24-3. Based on the acquisition function, use the L-BFGS-B algorithm to solve the optimal parameters of the Transformer model: ； Among them, represents the next optimal parameter point found by the optimization algorithm; represents finding the parameter that maximizes ; represents the DC bias current when the optical device is operating; represents the supply voltage of the optical device; represents the constraint condition; S24-4. Obtain new samples based on the optimal parameter points, update the basic training set, and obtain the updated training set: ； Among them, represents the basic training set; represents the updated training set; represents the true value obtained by actually measuring the optimal parameter point ; S24-5, based on the total uncertainty Obtain the standard deviation threshold , if detected , then retrain the Transformer model using the updated training set to construct a defect detection model; Among them, represents the predicted mean of the Transformer model at the optimal parameter point ; represents the standard deviation threshold.

9. An optical device defect detection system, characterized in that, The system includes: A multi-modal data acquisition module for acquiring the performance parameters and multi-angle appearance images of the optical device to be detected; the performance parameters include optical power, spectral flatness, current, voltage, and impedance; A multi-modal data processing module for first performing normalization processing on the acquired performance parameters and appearance images, then performing feature encoding to obtain the image feature vector and the parameter feature vector, and finally performing concatenation processing on the image feature vector and the parameter feature vector; A Transformer defect detection module for performing feature extraction on the concatenation processing result and outputting the defect detection result; An active learning and optimization module for optimizing the parameters of the Transformer model using the Bayesian optimization engine and triggering the retraining of the Transformer model.

Images