Network structure and detection method for surface defect detection of optical communication device based on twin architecture

Abstract

The application discloses a kind of based on twin architecture optical communication device surface defect detection network structure and detection method, belong to image processing related technical field.The surface defect detection network structure includes feature extraction encoder and feature fusion decoder;Feature extraction encoder is composed of twin ResNet residual network, non-defect feature matching elimination module and defect feature enhancement module;Defect feature enhancement module is composed of Transform coding decoder and convolution triplet attention module;Feature fusion decoder uses multilayer perception module.Improved ResNet residual network is used to extract the multi-scale image features of the sample image to be tested and the template sample image simultaneously, eliminate non-defect features, obtain difference feature maps, and output the segmentation results of the final defect area after defect feature enhancement and multilayer perception processing.The application realizes accurate and rapid segmentation of the surface defect area of the optical communication device, significantly improves the efficiency and accuracy of the surface defect segmentation and defect detection of the optical communication device.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a network structure and detection method for surface defects in optical communication devices based on a twin architecture. Background Technology

[0002] Optical communication devices are the core components of optical communication equipment, primarily used for converting between photoelectric signals. They are widely used in industrial control systems, the internet, and mobile communications. The quality of optical communication devices directly affects the overall performance of optical communication equipment. Surface defect detection is a crucial step in the automated manufacturing process of optical communication devices. Due to their complex manufacturing processes, optical communication devices are highly susceptible to defects caused by factors such as production equipment, processing techniques, and human error during production, leading to substandard quality. Therefore, it is essential to use visual inspection methods to filter out substandard optical communication devices.

[0003] Optical communication devices are riddled with defects of various types and forms, making segmentation extremely difficult. During production, they are prone to common defects with readily apparent characteristics, such as contamination, scratches, dents, and breakage, as well as unique defects like missing or incorrect solder joints and wires. Since the characteristics of solder joints and wires themselves are not considered defects, only by using a flawless optical communication device of the same model as a template and identifying differences between the two samples can the presence of missing or incorrect solder joints and wires in the device under test be determined. For these unique defects, single-unit defect segmentation networks such as U-Net and SegFormer, trained solely on defect samples, are ineffective. To accurately detect potential defects such as missing or incorrect wires in optical communication devices, a change detection network targeting the differences between two input images is required.

[0004] Currently, ChangeFormer is one of the highest-performing change detection models. It is a Siamese network based on the Transformer architecture. ChangeFormer utilizes the self-attention mechanism of Transformer to capture long-range dependencies between image features; it employs a parameter-sharing Siamese network structure to process input images, enabling the model to learn the differences between two input images. This method provides a detection approach for anomalies such as "missing lines" and "misaligned lines" in optical communication devices. However, in addition to the aforementioned anomalies, the optical communication device under test may also have relative positional offsets of patch components and solder joints relative to the template optical communication device sample. Such positional offsets do not affect the working performance of the optical communication device and are not considered defects or anomalies.

[0005] However, the existing ChangeFormer change detection network, through direct change detection logic, will also report such positional offsets as defects, ultimately causing large-scale false detections on the production line and seriously reducing the detection accuracy of optical communication device production lines. Summary of the Invention

[0006] To address the shortcomings of related technologies, the present invention aims to provide a network structure and detection method for surface defects in optical communication devices based on a twin architecture. This invention addresses the problem that existing detection technologies misjudge features caused by relative positional shifts that are not defects, leading to false alarms on the production line and an inability to accurately detect surface defects that affect the working performance of optical communication devices.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a surface defect network structure for optical communication devices based on a twin architecture, comprising: a feature extraction encoder and a feature fusion decoder;

[0008] The feature extraction encoder includes a Siamese ResNet residual network, a non-defect feature matching and elimination module, and a defect feature enhancement module;

[0009] The Siamese ResNet residual network is used to extract multi-scale features from the input test sample image and template sample image, and output five sets of feature maps at different scales respectively; the test sample image and template sample image are obtained by concatenating the coaxial light and low ring light images of the template sample and the test sample through the channel dimension.

[0010] The non-defect feature matching and elimination module is used to set the feature intensity of the corresponding position of the matching point in the two feature maps of the same scale in the four sets of feature maps that are not at the maximum scale to 0, so as to obtain the difference feature map at the corresponding scale; the non-defect feature matching and elimination module is also used to subtract the feature map of the maximum scale in the test sample and the template sample to obtain the difference feature map.

[0011] The defect feature enhancement module is used to enhance the defect features of the difference feature map and output the defect-enhanced difference feature map.

[0012] The feature fusion decoder is used to unify the scale, stitch together and compress the channels of five different scale feature maps to achieve multi-scale feature fusion and output the defect region segmentation result.

[0013] Optionally, the Siamese ResNet residual network comprises two ResNet residual networks with identical structures and shared weights; the Siamese ResNet residual network eliminates the max pooling layer before the output of the first ResNet layer.

[0014] Optionally, the five sets of differential feature maps are 64-dimensional × 128 pixels × 128 pixels, 64-dimensional × 64 pixels × 64 pixels, 128-dimensional × 32 pixels × 32 pixels, 256-dimensional × 16 pixels × 16 pixels, and 512-dimensional × 8 pixels × 8 pixels.

[0015] Optionally, the non-defect feature matching and elimination module includes a feature sequence generation submodule, a lightweight Transformer architecture, a feature matching submodule, and a matching feature elimination submodule;

[0016] The input to the non-defect feature matching and elimination module is four sets of non-maximum scale feature maps of C-dimensional × W pixels × W pixels in the test sample and template sample.

[0017] The feature sequence generation submodule is used to flatten the two input feature maps of the same scale into a length of W. 2 The characteristic sequence;

[0018] The Transformer architecture is used to process the feature sequences of the test sample feature map and the template sample feature map respectively using self-attention mechanism and cross-attention mechanism, and the process is repeated N times in sequence to obtain the processed feature sequence.

[0019] The feature matching submodule is used to construct a similarity matrix based on the similarity of the feature sequences of the test sample and the template sample; after the similarity matrix is ​​normalized, matching points with high similarity are obtained by threshold screening, and then matching points with high confidence are selected from the matching points with high similarity according to the principle of mutual nearest neighbors, and the matching point index set of the feature maps of the test sample and the template sample is obtained according to the final matching result.

[0020] The matching feature elimination submodule creates a W-pixel × W-pixel mask matrix for the test sample and a template sample mask matrix of the same size as the feature map based on the matching point index set. The values ​​at the corresponding matching point index positions in the mask matrix are set to 0, and the remaining positions are set to 1. The mask matrix for the test sample and the template sample mask matrix are multiplied by the feature vectors at the corresponding positions of the input test sample feature map and template sample feature map, respectively, to obtain the test sample feature map and template sample feature map with successfully eliminated non-defective features. The difference features are then calculated to obtain the difference feature map at the corresponding scale.

[0021] Optionally, different scales of feature maps correspond to different N values; the smaller the scale of the feature map, the larger N is.

[0022] Optionally, the defect feature enhancement module consists of a Transformer codec and a convolutional triplet attention module;

[0023] The Transformer codec is used to enhance the defect features of five differential feature maps at different scales, and output five differential feature maps at different scales containing global context information and defect feature enhancement.

[0024] The convolutional triplet attention module is used to calculate the weighted attention weights of multiple dimensions of the enhanced differential feature map, and apply the weighted attention weights to the differential feature maps of the corresponding dimensions respectively, and output the defect-enhanced differential feature map.

[0025] Optionally, the multilayer perceptron module includes a first multilayer perceptron, an upsampling submodule, a channel stitching submodule, a second multilayer perceptron, a two-dimensional transposed convolutional layer, and a segmentation head;

[0026] The first multilayer perceptron is used to unify the channels of five differential feature maps at different scales into 64 dimensions, resulting in five differential feature maps at different scales.

[0027] The upsampling submodule is used to upsample four difference feature maps with a scale of 64D×64Pixel×64Pixel, 64D×32Pixel×32Pixel, 64D×16Pixel×16Pixel, and 64D×8Pixel×8Pixel to a size of 128Pixel×128Pixel through bicubic interpolation, resulting in five difference feature maps with a scale of 64D×128Pixel×128Pixel.

[0028] The channel stitching submodule is used to upsample the five difference feature maps and stitch them together in the channel dimension to obtain a difference feature map with a scale of 320 dimensions × 128 pixels × 128 pixels.

[0029] The second multilayer perceptron is used to compress the number of channels in the feature map from 320 to 64, enabling multi-scale feature fusion.

[0030] The two-dimensional transposed convolutional layer upsamples the fused feature map and adjusts its size to 256 pixels × 256 pixels, resulting in a feature map with a scale of 64 dimensions × 256 pixels × 256 pixels.

[0031] The segmentation head is used to predict the category of "defect" and "non-defect" for each pixel in the input 64-dimensional × 256-pixel × 256-pixel feature map, and obtain a binary defect segmentation result image with a size of 256 pixels × 256 pixels; wherein, the black area with a pixel value of 0 represents the normal area, and the white area with a pixel value of 255 represents the defect area.

[0032] In a second aspect, the present invention also provides a method for detecting surface defects in optical communication devices based on a twin architecture, using a network structure as described in any one of the first aspects, including:

[0033] Images of the template sample and the test sample under coaxial light and low ring light are acquired, and then stitched together along the channel dimension to obtain the test sample image and the template sample image, respectively.

[0034] The template sample image and the sample image to be tested are input into the surface defect detection network of optical communication devices based on twin architecture, and the defect region segmentation result is output.

[0035] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0036] 1. This invention provides a surface defect detection network for optical communication devices based on a twin architecture. Utilizing a change detection approach, a dataset is constructed consisting of test samples and template samples under both coaxial and low-ring lighting conditions. An improved ResNet residual network is used to simultaneously extract multi-scale image features from both samples, eliminating non-defect features in both the test and template feature maps. Through change detection, the difference features between the test and template samples are obtained and considered as defect features. Five difference feature maps at different scales undergo defect feature enhancement processing before being input into a multilayer perceptron to output the final segmentation result of the defect region. This defect detection network achieves accurate segmentation of various types and shapes of defects in optical communication devices, including surface-mount components and solder joints with relative positional offsets. It also weakens non-defect features caused by relative positional offsets between the test and template samples, achieving precise and rapid segmentation of surface defect regions in optical communication devices. This significantly improves the efficiency and accuracy of surface defect segmentation and detection in optical communication devices.

[0037] 2. This invention provides a surface defect detection network for optical communication devices based on a twin architecture. It incorporates a Feature Matching for Non-defect Elimination (FMNE) module, which employs a lightweight Transformer architecture. This reduces computational resource consumption and, by generating a mask matrix, mitigates the impact of non-defect features commonly found in optical communication devices, such as solder joints, wire bonding, and relative offsets of surface-mount components, on production line inspection. The segmentation and detection accuracy on optical communication device datasets is significantly higher than existing conventional segmentation models. During the extraction of difference features between the test sample and the template sample, the FMNE module eliminates non-defect features from the difference features. A defect feature enhancement module accurately captures both large-span and small-area defects in the samples. A lightweight multilayer perceptron is used to fuse multi-scale difference features, and finally, the segmentation head obtains the final defect segmentation result.

[0038] 3. This invention provides a surface defect detection network for optical communication devices based on a twin architecture. It employs a defect feature enhancement module for optical communication devices, which integrates low-level detail information and high-level semantic information in the difference feature map extracted by the twin network. At the same time, it captures important cross-dimensional feature information in the feature map, which can generate feature maps with richer information and more distinct defect features, thus improving the defect segmentation accuracy and performance of the network designed in this invention.

[0039] 4. This invention provides a surface defect detection network for optical communication devices based on a twin architecture. The surface defect detection network fuses feature information from five feature maps at different scales using a lightweight multilayer perceptron. Compared with conventional segmentation networks that only use upsampling of the highest-dimensional feature map to obtain the final segmentation result, the network designed in this invention significantly improves the defect segmentation accuracy.

[0040] 5. This invention provides a surface defect detection method for optical communication devices based on a twin architecture. It utilizes an improved ResNet residual network to simultaneously extract multi-scale image features from two samples, eliminating non-defect features from both the test sample feature map and the template sample feature map to obtain differential features. These five differential feature maps at different scales are then processed by defect feature enhancement and a multilayer perceptron to obtain the final segmentation result of the defect region, thus achieving surface defect detection. This scheme can accurately segment and detect various types of defects, significantly reducing the false alarm rate on optical communication device production lines. It achieves precise and rapid segmentation and detection of surface defect regions in optical communication devices, significantly improving the efficiency and accuracy of surface defect segmentation and detection. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a surface defect detection network for optical communication devices based on a twin architecture, provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the non-defect feature matching and elimination module provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the lightweight Transformer in the non-defect feature matching and elimination module provided in this embodiment of the invention;

[0044] Figure 4 This is a schematic diagram illustrating the working principle of the mask matrix in the non-defect feature matching and elimination module provided in this embodiment of the invention.

[0045] Figure 5 These are the test samples and defect segmentation results provided in the embodiments of the present invention;

[0046] Figure 6This is a schematic diagram of the relative position offset of the optical communication device provided in an embodiment of the present invention;

[0047] Figure 7 This is a visualization of the non-defect feature matching and elimination module provided in this embodiment of the invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0050] Example 1

[0051] A surface defect detection network structure for optical communication devices based on a twin architecture includes: a feature extraction encoder and a feature fusion decoder;

[0052] The feature extraction encoder includes a Siamese ResNet residual network, a non-defect feature matching and elimination module, and a defect feature enhancement module;

[0053] The Siamese ResNet residual network is used to extract multi-scale features from the input test sample image and template sample image, and output five sets of feature maps at different scales respectively; the test sample image and template sample image are obtained by concatenating the coaxial light and low ring light images of the template sample and the test sample through the channel dimension.

[0054] The non-defect feature matching and elimination module is used to set the feature intensity of the corresponding position of the matching point in the two feature maps of the same scale in the four sets of feature maps that are not at the maximum scale to 0, so as to obtain the difference feature map at the corresponding scale; the non-defect feature matching and elimination module is also used to subtract the feature map of the maximum scale in the test sample and the template sample to obtain the difference feature map.

[0055] The defect feature enhancement module is used to enhance the defect features of the difference feature map and output the defect-enhanced difference feature map.

[0056] The feature fusion decoder is used to unify the scale, stitch together and compress the channels of five different scale feature maps to achieve multi-scale feature fusion and output the defect region segmentation result.

[0057] Given the problems with existing technologies, there is an urgent need to develop a defect detection method for optical communication devices that can not only cope with the diverse types and shapes of surface defects in optical communication devices, but also reduce the impact of relative position offset on defect detection tasks, so as to achieve accurate detection of surface defects of various types of optical communication devices.

[0058] like Figure 1 As shown in the figure, this invention discloses a surface defect detection network for optical communication devices based on a twin architecture. This network is used to detect and segment a wide variety of surface defects in optical communication devices, while simultaneously mitigating non-defect features caused by relative positional shifts between the test sample and the template sample. This avoids large-scale false alarms on the production line, achieving accurate and rapid segmentation of surface defect regions in optical communication devices, significantly improving the efficiency and accuracy of defect segmentation. By employing a change detection approach, the designed network structure learns how to identify differences between the template sample image and the test sample image, treating these differences as defect features. The main challenge of change detection lies in the fact that, in addition to defect features, there are also relative positional shifts between components such as solder joints and weld lines between the template sample image and the test sample image.

[0059] The defect detection network feature extraction encoder consists of an improved Siamese ResNet residual network, a non-defect feature matching and elimination module, and a defect feature enhancement module. The five sets of differential feature maps at different scales are 64D×128 pixels×128 pixels, 64D×64 pixels×64 pixels, 128D×32 pixels×32 pixels, 256D×16 pixels×16 pixels, and 512D×8 pixels×8 pixels.

[0060] The Siamese ResNet residual network in this embodiment comprises two ResNet residual networks with identical structures and shared weights. Based on this, the present invention eliminates the max-pooling layer before the first output layer of the ResNet residual network, adjusting the scale of the first output feature map from 64D×64 pixels×64 pixels to 64D×128 pixels×128 pixels. Simultaneously, to reduce the number of network parameters and make the network as lightweight as possible, the present invention replaces the standard convolutional layers in ResNet with depthwise separable convolutions, while retaining residual connections. The embodiment in this application uses the ResNet-34 network, and the scale of the five sets of feature maps at different scales can vary with the number of layers in the ResNet variant network. In alternative embodiments, variants of ResNet networks such as ResNet-18, ResNet-50, and ResNet-101 can also be used.

[0061] The network designed in this embodiment first extracts multi-scale features from the test sample image and the template sample image; then, it inputs four sets of feature maps with scales of 64D×64Pixel×64Pixel, 128D×32Pixel×32Pixel, 256D×16Pixel×16Pixel, and 512D×8Pixel×8Pixel from the multi-scale feature maps of the two samples into the network. Figure 2 and Figure 3 The non-defect feature matching and elimination module shown performs non-defect feature matching and elimination, weakening the non-defect features caused by the relative positional offset between the test sample and the template sample, and obtaining the difference feature map at the corresponding scale. Then, a direct subtraction operation is used to obtain the difference features of the test sample and the template sample at a scale of 64D × 128 pixels × 128 pixels. This difference feature map, along with the four difference feature maps obtained from the non-defect feature matching and elimination module, are input into the defect feature enhancement module to highlight the defect features. Finally, the feature maps at the five scales after defect feature enhancement are input into a lightweight feature fusion decoder, outputting the final segmentation result of the defect region.

[0062] The template sample image, which is stitched together from images under coaxial light and low ring light in the channel dimension, and the test sample image are input into the surface defect detection network of optical communication devices based on the twin architecture. The improved twin ResNet backbone network simultaneously extracts multi-scale features of the template sample and the test sample, and outputs five sets of feature maps at different scales, namely 64-dimensional × 128 pixels × 128 pixels, 64-dimensional × 64 pixels × 64 pixels, 128-dimensional × 32 pixels × 32 pixels, 256-dimensional × 16 pixels × 16 pixels, and 512-dimensional × 8 pixels × 8 pixels.

[0063] Optionally, the non-defect feature matching and elimination module includes a feature sequence generation submodule, a lightweight Transformer architecture, a feature matching submodule, and a matching feature elimination submodule;

[0064] The input to the non-defect feature matching and elimination module is four sets of non-maximum scale feature maps of C-dimensional × W pixels × W pixels in the test sample and template sample.

[0065] The feature sequence generation submodule is used to flatten the two input feature maps of the same scale into a length of W. 2 The characteristic sequence;

[0066] The Transformer architecture is used to process the feature sequences of the test sample feature map and the template sample feature map respectively using self-attention mechanism and cross-attention mechanism, and the process is repeated N times in sequence to obtain the processed feature sequence.

[0067] The feature matching submodule is used to construct a similarity matrix based on the similarity of the feature sequences of the test sample and the template sample; after the similarity matrix is ​​normalized, matching points with high similarity are obtained by threshold screening, and then matching points with high confidence are selected from the matching points with high similarity according to the principle of mutual nearest neighbors, and the matching point index set of the feature maps of the test sample and the template sample is obtained according to the final matching result.

[0068] The matching feature elimination submodule creates a W-pixel × W-pixel mask matrix for the test sample and a template sample mask matrix of the same size as the feature map based on the matching point index set. The values ​​at the corresponding matching point index positions in the mask matrix are set to 0, and the remaining positions are set to 1. The mask matrix for the test sample and the template sample mask matrix are multiplied by the feature vectors at the corresponding positions of the input test sample feature map and template sample feature map, respectively, to obtain the test sample feature map and template sample feature map with successfully eliminated non-defective features. The difference features are then calculated to obtain the difference feature map at the corresponding scale.

[0069] The four sets of feature maps obtained from the test sample and template sample, with scales of 512D×8Pixel×8Pixel, 256D×16Pixel×16Pixel, 128D×32Pixel×32Pixel, and 64D×64Pixel×64Pixel, are respectively input into the following... Figure 2 In the Feature Matching for Non-defect Elimination (FMNE) module shown, two feature maps of the same scale are flattened into sequences, and then... Figure 3 The lightweight Transformer shown obtains the feature sequences of the test sample and template sample after processing with self-attention and cross-attention mechanisms. Treating the feature sequences as feature vectors, the similarity between feature vectors is obtained through vector inner product operations, and a similarity matrix is ​​constructed. Based on the similarity matrix and the nearest neighbor matching principle, a set of matching point indices for the two feature sequences is obtained, and then a mask matrix for the test sample and a mask matrix for the template sample, with the same size as the feature map of the sample, are created. Figure 4 As shown, the mask matrix is ​​multiplied by the feature vectors at corresponding positions in the feature map, and the feature intensity at the matching point is set to 0. This eliminates the features of successful matching between the test sample and the template sample. Then, the difference features between the test sample and the template sample at corresponding scales after eliminating non-defect features are calculated, resulting in four difference feature maps with scales of 512D×8 pixels×8 pixels, 256D×16 pixels×16 pixels, 128D×32 pixels×32 pixels, and 64D×64 pixels×64 pixels. These difference feature maps have largely eliminated the matching image features between the test sample and the template sample, while retaining the defect features on the test sample.

[0070] Optionally, different scales of feature maps correspond to different N values; the smaller the scale of the feature map, the larger N is. In this embodiment, when the feature map scale is 512 dimensions × 8 pixels × 8 pixels, N is 4; when the feature map scale is 256 dimensions × 16 pixels × 16 pixels, 128 dimensions × 32 pixels × 32 pixels, or 64 dimensions × 64 pixels × 64 pixels, N is 2.

[0071] Utilize Figure 3 The lightweight Transformer shown applies a self-attention mechanism to both the feature maps of the test sample and the template sample, and then applies a cross-attention mechanism to both. This self-attention and cross-attention mechanism is then repeated N times in sequence. Through the application of self-attention and cross-attention mechanisms, an enhanced feature sequence is obtained that incorporates relative positional information and global contextual information from the test sample and the template sample.

[0072] Specifically, such as Figure 3 As shown, the lightweight Transformer first obtains the query vector from feature sequence A, and then obtains the key vector and value vector from feature sequence B. Taking the feature sequence of the test sample as an example, when performing self-attention on the feature sequence of the test sample, feature sequence A and feature sequence B are the same, both being the feature sequence of the test sample. When performing cross-attention on the feature sequence of the test sample, feature sequence A is the feature sequence of the test sample, and feature sequence B is the feature sequence of the template sample.

[0073] The query vector, key vector, and value vector are generated using the following formula:

[0074] Q = UW Q K = RW K V = RW V

[0075] Where U and R represent, respectively, Figure 3 The feature sequences A and B shown are W. Q W K W V These are the learnable weights for feature transformation.

[0076] The self-attention mechanism and cross-attention operation of the lightweight Transformer are performed according to the following formula:

[0077]

[0078] in, This represents the template sample feature sequence at scale i after adding positional encoding, using... Atten represents the feature sequence of the test sample at scale i after adding positional encoding. self Representing the self-attention mechanism, Attencross Represents the cross-attention mechanism. This represents the template sample feature map at scale i after undergoing self-attention and cross-attention mechanisms. This represents the feature map of the sample under test at scale i after undergoing self-attention and cross-attention mechanisms.

[0079] The feature sequences of the test sample and template sample are treated as feature vectors, and an inner product operation is performed between the vectors to obtain a similarity matrix. Then, the Softmax operator is applied along both the row and column dimensions to normalize the similarity matrix. High-similarity matching points are obtained through threshold filtering, and then high-confidence matching points are further filtered according to the nearest neighbor principle. The indices of each matching point in the feature maps of the test sample and template sample are obtained based on the final matching results.

[0080] like Figure 4 As shown, a W-pixel × W-pixel mask matrix for the test sample and a template sample mask matrix of the same size as the feature map are created. Successfully matched non-defect features are eliminated, and the difference features are calculated to obtain a difference feature map at the corresponding scale. This difference feature map is used as the output difference feature map of the non-defect feature matching and elimination module, as shown below. Figure 7 As shown.

[0081] Optionally, the defect feature enhancement module consists of a Transformer codec and a convolutional triplet attention module;

[0082] The Transformer codec is used to enhance the defect features of five differential feature maps at different scales, and output five differential feature maps at different scales containing global context information and defect feature enhancement.

[0083] The convolutional triplet attention module is used to calculate the weighted attention weights of multiple dimensions of the enhanced differential feature map, and apply the weighted attention weights to the differential feature maps of the corresponding dimensions respectively, and output the defect-enhanced differential feature map.

[0084] Defect feature enhancement includes the following steps: The differential feature map containing defect features, obtained by the non-defect feature matching and elimination module, is input into the Transformer codec in the defect feature enhancement module. The encoder of the Transformer codec is responsible for capturing high-level semantic information in the feature map, while the decoder uses this information to reconstruct a feature map that is richer in information and has more prominent defect features. The feature map processed by the Transformer codec integrates global contextual information of the feature map, achieving more accurate and complete feature extraction, especially for large-area, long-distance defects, thus improving the accuracy and performance of the defect detection network for optical communication devices. Five feature maps with scales of 64D×128 pixels×128 pixels, 64D×64 pixels×64 pixels, 128D×32 pixels×32 pixels, 256D×16 pixels×16 pixels, and 512D×8 pixels×8 pixels, output from the Transformer decoder, are input into the Convolutional Triplet Attention Module (CTAM) in the defect feature enhancement module. The CTAM captures cross-dimensional interaction information between channel C and spatial dimensions H (height) or W (width), solving the problem of the disappearance of interdependence between channel dimensions and spatial dimensions, effectively enhancing the surface defect features of optical communication devices, and obtaining five feature maps with scales of 64D×128 pixels×128 pixels, 64D×64 pixels×64 pixels, 128D×32 pixels×32 pixels, 256D×16 pixels×16 pixels, and 512D×8 pixels×8 pixels.

[0085] Optionally, the multilayer perceptron module includes a first multilayer perceptron, an upsampling submodule, a channel stitching submodule, a second multilayer perceptron, a two-dimensional transposed convolutional layer, and a segmentation head;

[0086] The first multilayer perceptron is used to unify the channels of five differential feature maps at different scales into 64 dimensions, resulting in five differential feature maps at different scales.

[0087] The upsampling submodule is used to upsample four difference feature maps with a scale of 64D×64Pixel×64Pixel, 64D×32Pixel×32Pixel, 64D×16Pixel×16Pixel, and 64D×8Pixel×8Pixel to a size of 128Pixel×128Pixel through bicubic interpolation, resulting in five difference feature maps with a scale of 64D×128Pixel×128Pixel.

[0088] The channel stitching submodule is used to upsample the five difference feature maps and stitch them together in the channel dimension to obtain a difference feature map with a scale of 320 dimensions × 128 pixels × 128 pixels.

[0089] The second multilayer perceptron is used to compress the number of channels in the feature map from 320 to 64, enabling multi-scale feature fusion.

[0090] The two-dimensional transposed convolutional layer upsamples the fused feature map and adjusts its size to 256 pixels × 256 pixels, resulting in a feature map with a scale of 64 dimensions × 256 pixels × 256 pixels.

[0091] The segmentation head is used to predict the category of "defect" and "non-defect" for each pixel in the input 64-dimensional × 256-pixel × 256-pixel feature map, and obtain a binary defect segmentation result image with a size of 256 pixels × 256 pixels; wherein, the black area with a pixel value of 0 represents the normal area, and the white area with a pixel value of 255 represents the defect area.

[0092] In one specific embodiment, the following steps are included:

[0093] A dataset for surface defect segmentation of optical communication devices was obtained. This embodiment focuses on the image acquisition stage of the surface defect segmentation process for optical communication devices. A total of 600 defect sample images were collected from 15 models of optical communication devices, representing 4 types of defects. Data augmentation was performed on each type of defect sample, increasing the dataset size to 875 images. The surface defects in the dataset include categories such as base damage, base scratches, component contamination and breakage, and multiple / incorrect / incomplete solder lines. The sample images are in PNG format, with each image measuring 256 pixels × 256 pixels. Simultaneously, sample images of defect-free optical communication devices of the same model were collected as template samples for that model. The dataset size of template sample images also reached 875 images, corresponding one-to-one with the test sample images. The images were then divided into training, testing, and validation sets in a 7:2:1 ratio.

[0094] A surface defect segmentation model for optical communication devices based on a twin architecture is constructed. In this embodiment, the twin backbone network model used is an improved ResNet-34 residual network. The defect segmentation model designed in this invention can also use networks such as ResNet-18, ResNet-50, and ResNet-101 as twin backbone networks.

[0095] For each pixel in the sample image to be tested, since the output of the neural network is either defective or non-defective, it can be equivalent to a binary classification network. Therefore, this embodiment selects the binary classification cross-entropy loss function as the loss function of the defect detection network:

[0096]

[0097] Where: P 1i T represents the predicted probability of each pixel being a defect. iis the true label for each pixel, with a value of 0 or 255, representing non-defect and defect, respectively. N is the total number of pixels in the image under test. The cross-entropy loss function aims to make the defect prediction probability of each pixel as close as possible to the actual label probability.

[0098] In this embodiment, the AdamM algorithm is used as the model optimizer. The surface defect detection network is trained for 300 rounds with an initial learning rate of 0.0001.

[0099] In this embodiment, the backbone network of the surface defect segmentation model for optical communication devices based on a twin architecture is a ResNet-34 residual network, where standard convolutions are replaced with depthwise separable convolutions, and the activation function after each convolutional layer is the ReLU function. Fully connected layers are responsible for determining whether each pixel is a defect, acting as a classifier. Apart from the fully connected layers, the remaining convolutional parts are responsible for feature extraction.

[0100] Write a Python program to randomly sort the training samples, distribute them evenly, and train the optical communication device surface defect detection network using a batch size of 16 (unit: sheets) in the optical communication device dataset. Save the weight parameters of the optimized network model.

[0101] Specifically, in this embodiment, training images are input into the model in batches. After the sum of gradient descent values ​​for all samples in a batch is calculated, a weight update is performed until all batches are updated. The trained model is evaluated using a validation set to obtain the network validation loss value and calculate the IoU (Intersection over Union). If the IoU is greater than the current maximum IoU, the network model parameters and weights for that round are stored, and the next iteration begins; if it is less than the maximum IoU, the next iteration begins directly. The number of iterations for the network model is set to 300, and training ends after the required number of iterations is completed. The optimal network model is obtained and named FSNet_best_model.

[0102] The network weights corresponding to FSNet_best_model were loaded, and the network performance was tested on the test set to evaluate the defect segmentation accuracy of the defect segmentation model. The IoU (Intersection over Union) of the defect segmentation model was calculated. The final test set accuracy was 0.9472, recall was 0.8692, F1 score was 0.9065, and IoU was 0.8290, achieving a very accurate surface defect segmentation effect for optical communication devices.

[0103] In this embodiment of the invention, the test sample image of an optical communication device, formed by stitching images of coaxial light and low-ring light along the channel dimension, and the template sample image are input into a surface defect detection network. An improved ResNet residual network is used to simultaneously extract multi-scale image features from both samples, eliminating non-defect features in both the test sample feature map and the template sample feature map to obtain a difference feature map. Five difference feature maps at different scales are then processed for defect feature enhancement before being input into a lightweight multilayer perceptron to output the final segmentation result of the defect region. This solves the technical problem of existing detection technologies misjudging features due to relative positional offsets that are not defects, leading to false alarms on the production line and an inability to accurately detect surface defects affecting the working performance of optical communication devices. It achieves accurate segmentation of various defects, significantly reduces the false alarm rate on the optical communication device production line, and enables precise and rapid segmentation of surface defect regions of optical communication devices, significantly improving the efficiency and accuracy of surface defect segmentation and defect detection for optical communication devices.

[0104] Example 2

[0105] A method for detecting surface defects in optical communication devices based on a twin architecture, using the network structure described in any one of Embodiment 1, includes:

[0106] S1. Obtain images of the template sample and the test sample under coaxial light and low ring light illumination, and then stitch them together along the channel dimension to obtain the test sample image and the template sample image respectively.

[0107] S2. Input the template sample image and the sample image to be tested into the surface defect detection network of optical communication devices based on twin architecture, and output the defect region segmentation result.

[0108] This invention provides a method for detecting surface defects in optical communication devices based on a twin architecture. It is implemented using a twin-architecture-based network structure for detecting surface defects in optical communication devices, achieving the same functionality and beneficial effects. This method can solve the problems of difficulty in accurately segmenting defects due to their diverse types and characteristics in optical communication device surface defect segmentation tasks, as well as the large-scale false detections on the production line caused by the relative positional misalignment of patch components and solder joints in the template sample image and the test sample image.

[0109] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A surface defect detection network structure for optical communication devices based on a twin architecture, characterized in that, include: Feature extraction encoder and feature fusion decoder; The feature extraction encoder includes a Siamese ResNet residual network, a non-defect feature matching and elimination module, and a defect feature enhancement module; The Siamese ResNet residual network is used to extract multi-scale features from the input test sample image and template sample image, and output five sets of feature maps at different scales respectively; the test sample image and template sample image are obtained by concatenating the coaxial light and low ring light images of the template sample and the test sample through the channel dimension. The non-defect feature matching and elimination module is used to set the feature intensity of the corresponding position of the matching point in the two feature maps of the same scale in the four sets of feature maps that are not at the maximum scale to 0, so as to obtain the difference feature map at the corresponding scale; the non-defect feature matching and elimination module is also used to subtract the feature map of the maximum scale in the test sample and the template sample to obtain the difference feature map. The defect feature enhancement module is used to enhance the defect features of the difference feature map and output the defect-enhanced difference feature map. The feature fusion decoder is used to unify the scale, stitch together and compress the channels of five different scale feature maps to achieve multi-scale feature fusion and output the defect region segmentation result.

2. The network structure as described in claim 1, characterized in that, The twin ResNet residual network comprises two ResNet residual networks with identical structures and shared weights; the twin ResNet residual network eliminates the max pooling layer before the output of the first ResNet layer.

3. The network structure as described in claim 1, characterized in that, The five sets of differential feature maps at different scales are 64-dimensional × 128 pixels × 128 pixels, 64-dimensional × 64 pixels × 64 pixels, 128-dimensional × 32 pixels × 32 pixels, 256-dimensional × 16 pixels × 16 pixels, and 512-dimensional × 8 pixels × 8 pixels.

4. The network structure as described in claim 1, characterized in that, The non-defect feature matching and elimination module includes a feature sequence generation submodule, a lightweight Transformer architecture, a feature matching submodule, and a matching feature elimination submodule. The input to the non-defect feature matching and elimination module is four sets of non-maximum scale feature maps of C-dimensional × W pixels × W pixels in the test sample and template sample. The feature sequence generation submodule is used to flatten the two input feature maps of the same scale into a length of W. 2 The characteristic sequence; The Transformer architecture is used to process the feature sequences of the test sample feature map and the template sample feature map respectively using self-attention mechanism and cross-attention mechanism, and the process is repeated N times in sequence to obtain the processed feature sequence. The feature matching submodule is used to construct a similarity matrix based on the similarity of the feature sequences of the sample to be tested and the template sample; After the similarity matrix is ​​normalized, matching points with high similarity are obtained by thresholding. Then, according to the principle of mutual nearest neighbors, matching points with high confidence are selected from the matching points with high similarity. Based on the final matching results, the matching point index set of the feature maps of the test sample and the template sample is obtained. The matching feature elimination submodule creates a W-pixel × W-pixel mask matrix for the test sample and a template sample mask matrix of the same size as the feature map based on the matching point index set. The values ​​at the corresponding matching point index positions in the mask matrix are set to 0, and the remaining positions are set to 1. The mask matrix for the test sample and the template sample mask matrix are multiplied by the feature vectors at the corresponding positions of the input test sample feature map and template sample feature map, respectively, to obtain the test sample feature map and template sample feature map with successfully eliminated non-defective features. The difference features are then calculated to obtain the difference feature map at the corresponding scale.

5. The network structure as described in claim 4, characterized in that, Different scales of feature maps correspond to different N values; the smaller the scale of the feature map, the larger N is.

6. The network structure as described in claim 1, characterized in that, The defect feature enhancement module consists of a Transformer encoder / decoder and a convolutional triplet attention module. The Transformer codec is used to enhance the defect features of five differential feature maps at different scales, and output five differential feature maps at different scales containing global context information and defect feature enhancement. The convolutional triplet attention module is used to calculate the weighted attention weights of multiple dimensions of the enhanced differential feature map, and apply the weighted attention weights to the differential feature maps of the corresponding dimensions respectively, and output the defect-enhanced differential feature map.

7. The network structure as described in claim 1, characterized in that, The multilayer perceptron module includes a first multilayer perceptron, an upsampling submodule, a channel splicing submodule, a second multilayer perceptron, a two-dimensional transposed convolutional layer, and a segmentation head; The first multilayer perceptron is used to unify the channels of five differential feature maps at different scales into 64 dimensions, resulting in five differential feature maps at different scales. The upsampling submodule is used to upsample four difference feature maps with a scale of 64D×64Pixel×64Pixel, 64D×32Pixel×32Pixel, 64D×16Pixel×16Pixel, and 64D×8Pixel×8Pixel to a size of 128Pixel×128Pixel through bicubic interpolation, resulting in five difference feature maps with a scale of 64D×128Pixel×128Pixel. The channel stitching submodule is used to upsample the five difference feature maps and stitch them together in the channel dimension to obtain a difference feature map with a scale of 320 dimensions × 128 pixels × 128 pixels. The second multilayer perceptron is used to compress the number of channels in the feature map from 320 to 64, enabling multi-scale feature fusion. The two-dimensional transposed convolutional layer upsamples the fused feature map and adjusts its size to 256 pixels × 256 pixels, resulting in a feature map with a scale of 64 dimensions × 256 pixels × 256 pixels. The segmentation head is used to predict the category of "defect" and "non-defect" for each pixel in the input 64-dimensional × 256-pixel × 256-pixel feature map, and obtain a binary defect segmentation result image with a size of 256 pixels × 256 pixels; wherein, the black area with a pixel value of 0 represents the normal area, and the white area with a pixel value of 255 represents the defect area.

8. A method for detecting surface defects in optical communication devices based on a twin architecture, using the network structure described in any one of claims 1-7, characterized in that, include: Images of the template sample and the test sample under coaxial light and low ring light are acquired, and then stitched together along the channel dimension to obtain the test sample image and the template sample image, respectively. The template sample image and the sample image to be tested are input into the surface defect detection network of optical communication devices based on twin architecture, and the defect region segmentation result is output.

Images