Optical fiber end face detection device, method and optical fiber end face detection model training method

Abstract

The application provides a fiber end face detection device, which comprises a spatial domain encoder, a frequency domain encoder, a decoder and a segmentation result image generation module. The application detects image defects at the pixel level, combines the respective advantages of the spatial domain image and the frequency domain image, and can detect obvious defects and non-obvious defects. In addition, the decoder performs splicing fusion and decoding on the spatial domain feature maps and the frequency domain feature maps at different depths, and the feature information loss is smaller than that obtained by decoding only the feature maps at the deepest position, so that the accuracy of fiber end face detection is higher. The application further discloses a fiber end face detection method and a fiber end face detection model training method.

Description

Technical Field

[0001] This invention relates to the field of optical fiber communication technology, specifically to an optical fiber end-face detection device, method, and optical fiber end-face detection model training method. Background Technology

[0002] As is well known, defects on the fiber optic end face (scratches, cracks, dirt, dust, etc.) increase the bit error rate of data in fiber optic transmission systems, leading to increased network retransmissions and reduced network communication quality. Therefore, fiber optic end face inspection is crucial in high-speed fiber optic link testing. In recent years, with the development of computer vision technology, defect detection systems based on traditional image processing and machine learning have been proposed, which can assist in the inspection of fiber optic end faces, significantly improving inspection efficiency. However, sometimes, because the defective areas on the fiber end face and the background texture of the fiber are visually very similar, fibers with defective end faces are mistakenly marked as qualified, causing significant harm to the transmission quality of the fiber optic communication system. Summary of the Invention

[0003] In view of this, it is necessary to propose a fiber optic end-face inspection device, which can at least improve the accuracy of fiber optic end-face defect detection.

[0004] Specifically, the fiber optic end-face detection device includes a spatial domain encoder, a frequency domain encoder, a decoder, and a segmentation result image generation module, wherein:

[0005] The spatial domain encoder is used to extract features and downsample the spatial domain image of the input optical fiber end face to obtain several spatial domain feature maps of different sizes, and each spatial domain feature map corresponds to a different depth of the spatial domain encoder.

[0006] The frequency domain encoder is used to extract features and downsample the frequency domain image of the input optical fiber end face to obtain several frequency domain feature maps of different sizes, and each frequency domain feature map corresponds to a different depth of the frequency domain encoder.

[0007] The decoder is used to perform feature fusion, feature extraction and upsampling on the embedded spatial domain feature map and frequency domain feature map at different depths, and output pixel-level classification result feature map;

[0008] The segmentation result image generation module is used to obtain the probability score of each pixel being predicted as a certain defect based on the pixel-level classification result feature map. The defect category with the highest probability score for each pixel is taken as the classification result of that pixel. The defect is segmented into pixels based on the classification results of all pixels to obtain the location and size of the fiber end face defect.

[0009] Furthermore, the optical fiber end face detection device also includes a spatial domain image generation module, which is used to extract the region of interest from the image of the optical fiber end face to be detected according to preset rules to obtain a spatial domain image.

[0010] Furthermore, the fiber end face detection device also includes a frequency domain image generation module, which performs Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

[0011] Furthermore, the spatial domain encoder, frequency domain encoder, and decoder adopt a neural network convolutional CNN model or a Swing Transformer model structure.

[0012] Furthermore, the spatial domain encoder and frequency domain encoder have the same structure, adopting the Swin Transformer model structure, and include at least one image block layer and several cascaded depth feature map output modules, wherein:

[0013] The image segmentation layer is used to divide the input spatial domain image or frequency domain image into several blocks of the same size.

[0014] The depth feature map output module includes a first depth feature map output module and several second depth feature map output modules, wherein:

[0015] The first deep feature map output module includes a linear embedding layer and a Swing Transformer block layer; the second deep feature map output module includes a block merging layer and a Swing Transformer block layer; wherein:

[0016] The linear embedding layer is used to perform linear transformation on the block map output by the image block layer, so that the feature map structure size output by the spatial domain encoder and the frequency domain encoder in the linear embedding layer is consistent; the block merging layer is used to reduce the resolution and increase the dimension of the feature map input to it; the Swing Transformer block layer is used to extract features from the feature map input to it according to preset rules.

[0017] Furthermore, the decoder includes several cascaded depth feature map decoding modules and a linear projection layer, wherein:

[0018] The deep feature map decoding module includes a fusion layer, a Swing Transformer block layer, and a block expansion layer, wherein:

[0019] The fusion layer is used to concatenate the input feature maps according to channels and positions, and then output them to the Swing Transformer block layer after linear transformation. The input feature maps include spatial domain feature maps, frequency domain feature maps, and feature maps output by the feature map decoding module of the previous depth. The Swing Transformer block layer is used to extract features from the input feature maps according to preset rules and then output them to the block expansion layer. The block expansion layer is used to increase the resolution and reduce the dimensionality of the input feature maps before outputting them.

[0020] The linear projection layer is used to map the feature map output by a depth feature map decoding module connected to it into a feature map with the same size as the original spatial domain image and a dimension equal to the number of defect categories.

[0021] Furthermore, the segmentation result image generation module is pre-set with a normalized exponential function (softmax) to obtain the probability score of each pixel being predicted as a certain defect. The content of the normalized exponential function (softmax) is as follows:

[0022]

[0023] Where z i Let C be the output value of a pixel in the i-th channel, and C be the number of categories.

[0024] On the other hand, the present invention also discloses a fiber optic end-face inspection method, which can at least improve the accuracy of fiber optic end-face defect detection.

[0025] The fiber optic end-face inspection method, utilizing the fiber optic end-face inspection device described above, includes at least the following steps:

[0026] The spatial domain image of the fiber end face to be detected is received, and several spatial domain feature maps of different sizes are obtained. Each spatial domain feature map corresponds to a different depth of the spatial domain encoder.

[0027] The frequency domain image of the fiber end face to be tested is received, and several frequency domain feature maps of different sizes are obtained. Each frequency domain feature map corresponds to a different depth of the frequency domain encoder.

[0028] The embedded spatial domain feature maps and frequency domain feature maps at different depths are fused, extracted and upsampled to output pixel-level classification result feature maps.

[0029] Based on the feature map of the pixel-level classification results, the probability score of each pixel being predicted as a certain defect is obtained. The defect category with the highest probability score for each pixel is taken as the classification result of that pixel. Based on the classification results of all pixels, the defect is segmented into pixels to obtain the location and size of the fiber end face defect.

[0030] Furthermore, the fiber end-face detection method further includes the following steps:

[0031] The region of interest is extracted from the image of the fiber end face to be inspected according to preset rules to obtain a spatial domain image;

[0032] Perform Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

[0033] Thirdly, the present invention also discloses a method for training a fiber optic end-face detection model, wherein the fiber optic end-face detection model includes a spatial domain encoder, a frequency domain encoder, and a decoder, and the training method includes:

[0034] The sample training set is fed into the fiber optic end-face detection model, and the model is optimized using a loss function. The sample training set includes several spatial domain images, frequency domain images, and defect category label images corresponding to each spatial domain image and frequency domain image. The loss function is:

[0035]

[0036] Where C represents the category label, P represents the number of pixels in the image, and g c The true label (0 or 1) representing the pixel belonging to category c, y c ε represents the predicted probability (in the range of 0 to 1) that a pixel belongs to class c, where ε is a constant to avoid the denominator being 0. Where f k This represents the frequency of category label k in the training samples.

[0037] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0038] This invention continuously downsamples the spatial and frequency domain images of the input fiber optic end face, obtaining several spatial and frequency domain feature maps of different sizes and at different depths in the spatial and frequency domain encoders (feature map resolution decreases, dimensionality increases). Then, at different depths in the decoder, the spatial and frequency domain feature maps at each depth are stitched together, fused, feature extracted, and upsampled to output pixel-level classification result feature maps. Based on these pixel-level classification result feature maps, a probability score is obtained for each pixel to be predicted as a certain type of defect. The defect category with the highest probability score for each pixel is taken as the classification result for that pixel. Based on the classification results of all pixels, defects are segmented into pixels to obtain the fiber optic end face defect detection result. This invention focuses on image defect detection at the pixel level, combining the advantages of both spatial and frequency domain images (i.e., spatial domain images retain the characteristics of the original image, while frequency domain images can obtain the texture information of the image), enabling the detection of both obvious and subtle defects. In addition, the shallower the spatial domain encoder and frequency domain encoder, the lower the dimension of the spatial domain feature map and the frequency domain feature map, but the more pixel details are preserved. Therefore, the decoder splices and fuses spatial domain feature maps and frequency domain feature maps at different depths and decodes them at different depths, resulting in less loss of feature information than decoding only the deepest feature map, thus ensuring higher accuracy of fiber end face detection. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of an optical fiber end face detection device in Embodiment 1 of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of an optical fiber end face detection device in Embodiment 2 of the present invention;

[0041] Figure 3 This is a schematic diagram of the depth feature map output module in Embodiment 2 of the present invention;

[0042] Figure 4 This is a schematic diagram of the structure of the depth feature map decoding module in Embodiment 2 of the present invention;

[0043] Figure 5 This is a flowchart of an optical fiber end-face detection method according to Embodiment 3 of the present invention;

[0044] Figure 6 This is a flowchart of an optical fiber end-face detection method in Embodiment 4 of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Example 1

[0047] In practical work, the inventors discovered that if detection is based solely on the spatial domain characteristics of the fiber end face, it is impossible to detect subtle defects because the characteristics of defects are not obvious in the spatial domain. If the frequency domain characteristics of the fiber end face are used for detection, edge detection and texture extraction of the fiber end face can be achieved, but false defects are easily detected, leading to serious misjudgments.

[0048] In view of this, the present invention discloses an optical fiber end-face detection device, such as... Figure 1 As shown, it includes a spatial domain encoder 10, a frequency domain encoder 20, a decoder 30, and a segmentation result image generation module 40, wherein:

[0049] The spatial domain encoder 10 is used to extract features and downsample the spatial domain image of the input optical fiber end face to obtain several spatial domain feature maps of different sizes, and each spatial domain feature map corresponds to a different depth of the spatial domain encoder 10.

[0050] Downsampling a spatial domain image means reducing the resolution and increasing the dimensionality of the resulting spatial domain feature map. This allows for the abstract representation of the input image using high-dimensional information, as higher dimensions contain more effective features. At the same time, it reduces the size of the feature map and minimizes redundant information.

[0051] Specifically, spatial domain images are generally three-dimensional images, which can be represented by 3×H×W, where 3 represents the dimension, H represents the height of the image, and W represents the width of the image.

[0052] The frequency domain encoder 20 is used to extract features and downsample the frequency domain image of the input optical fiber end face to obtain several frequency domain feature maps of different sizes, and each frequency domain feature map corresponds to a different depth of the frequency domain encoder 20.

[0053] Downsampling a frequency domain image means reducing the resolution and increasing the dimensionality of the resulting frequency domain feature map. This allows us to abstractly represent the input image using high-dimensional information, as higher dimensions contain more effective features. At the same time, it reduces the size of the feature map and minimizes redundant information.

[0054] Specifically, frequency domain images are generally four-dimensional images, which can be represented by 4×H×W, where 4 represents the dimension, H represents the height of the image, and W represents the width of the image.

[0055] In some embodiments, it is assumed that the spatial domain features generated by the spatial domain encoder 10 are The frequency domain features generated by the frequency domain encoder 20 are as follows: Among them, D, 2D, 4D, and 8D represent doubling the dimension of the feature map; the structural dimensions of E1 and F1, E2 and F2, E3 and F3, and E4 and F4 are the same.

[0056] The decoder 30 is used to perform feature fusion, feature extraction and upsampling (e.g., upsampling twice each time) on the embedded spatial domain feature map and frequency domain feature map at different depths, and then map the feature map output by the last upsampling through linear projection to a feature map with the same size as the original spatial domain image and the number of channels equal to the number of defect categories, and output a pixel-level classification result feature map C×H×W, where C is the number of channels, that is, the number of defect categories.

[0057] The segmentation result image generation module 40 is used to obtain the probability score of each pixel being predicted as a certain defect based on the pixel-level classification result feature map. The defect category with the highest probability score for each pixel is taken as the classification result for that pixel. That is, the fiber optic end-face inspection device predefines defect categories (such as scratches, specks, dirt, etc.) and defect severity (e.g., for scratches, obvious scratches, minor scratches) during the training phase. Then, defect segmentation is performed based on the classification results of all pixels, that is, the region defined as a defect is extracted. The location of this region is the location of the defect, and the number of pixels within this region is the size of the defect. The size of the defect can be represented by its perimeter or area. Obtaining the location and size of the defect enables defect detection of the fiber optic end face. In practical applications, defects need to be classified according to different customer needs. Some small defects are allowed (such as small specks on the edge of the product), while large defects (such as large dirt in the center of the fiber optic end face) are considered non-compliant.

[0058] In some embodiments, the feature map output by decoder 30 is passed through a normalized exponential function softmax layer to obtain a probability score for each pixel being predicted as a certain defect (including normal). The defect category with the highest probability score for each pixel is taken as the classification result for that pixel. Based on the classification results of all pixels, the defects are segmented to obtain a segmentation result map. H is the height of the image, W is the width of the image, and each pixel has C values. In actual use, the category represented by the channel with the largest value is taken as the final classification category. That is, it represents the classification category of a certain pixel.

[0059] The definition of the Softmax function is given below (taking the output of the i-th node as an example):

[0060]

[0061] Where z iLet C be the output value of the i-th node (i.e., the output value of a pixel in the i-th channel), and C be the number of output nodes (i.e., the number of channels), which is the number of categories. The Softmax function can be used to convert the output values ​​of multi-class classification into a probability distribution ranging from [0,1] to 1.

[0062] For example, during the training phase, the fiber optic end-face inspection device classifies defects into only two categories (no defects, defective), i.e., C=2. Therefore, each pixel in H×W has two channels. The image of the fiber end-face to be inspected is processed by the trained fiber optic end-face inspection device to obtain normalized values, such as 0.9 for channel one and 0.1 for channel two. At this point, channel one has the largest value. If channel one represents the no-defect category, then that pixel is recorded as no-defect.

[0063] For example, during the training phase, the fiber optic end-face inspection device classifies defects into four categories (no defects, scratches, specks, and dirt), i.e., C=4. Therefore, each pixel in H×W has four channels. The image of the fiber end-face to be inspected is processed by the trained fiber optic end-face inspection device to obtain normalized values, such as 0.2 for channel 1, 0.25 for channel 2, 0.3 for channel 3, and 0.25 for channel 4. At this point, channel 3 has the largest value. If channel 3 represents specks, then the defect recorded for that pixel is determined to be a speckle.

[0064] This invention continuously downsamples the spatial domain and frequency domain images of the input optical fiber end face to obtain several spatial domain feature maps and frequency domain feature maps of different sizes and located at different depths in the spatial domain encoder 10 and frequency domain encoder 20 (the feature map resolution is reduced, but the dimension is increased). Then, at different depths in the decoder 30, the spatial domain feature maps and frequency domain feature maps at each depth are stitched, fused, and upsampled to output pixel-level classification result feature maps. Then, based on the pixel-level classification result feature maps, the probability score of each pixel being predicted as a certain defect is obtained. The defect category with the highest probability score for each pixel is taken as the classification result for that pixel. Based on the classification results of all pixels, the defects are segmented into pixels to obtain the optical fiber end face defect detection result. This invention focuses on image defect detection at the pixel level, combining the advantages of both spatial domain and frequency domain images (i.e., spatial domain images retain the characteristics of the original image, and frequency domain images can obtain the texture information of the image), and can detect both obvious and inconspicuous defects. In addition, the shallower the spatial domain encoder 10 and frequency domain encoder 20 are, the lower the dimension of the spatial domain feature map and the frequency domain feature map are, but more pixel detail features are retained. Therefore, the decoder 30 splices and fuses spatial domain feature maps and frequency domain feature maps at different depths and decodes them at different depths, resulting in less loss of feature information than decoding only the deepest feature map, thus ensuring higher accuracy of fiber end face detection.

[0065] Example 2

[0066] In other embodiments, combined with Figure 2 As shown, the optical fiber end face detection device also includes a spatial domain image generation module 50, which is used to extract the region of interest from the image of the optical fiber end face to be detected according to preset rules to obtain a spatial domain image.

[0067] Specifically, the spatial domain image is a three-dimensional image, which can be represented by 3×H×W, where 3 represents the dimension, H represents the height of the image, and W represents the width of the image. In some embodiments, the spatial domain image can be obtained by extracting regions of interest (ROIs) from the fiber end face image to be detected according to preset rules. The preset rules include selecting or using methods such as setting thresholding on the spatial domain image to extract irregular shapes including points, lines, and surfaces. In this embodiment, the spatial domain image can be obtained by extracting regions of interest from the fiber end face image through threshold binarization.

[0068] In other embodiments, the fiber optic end face detection device further includes a frequency domain image generation module 60, which performs Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

[0069] Specifically, the frequency domain image is a four-dimensional image, which can be represented by 4×H×W, where 4 represents the dimension, H represents the height of the image, and W represents the width of the image. In some embodiments, the frequency domain image generation module can obtain it by performing a Haar wavelet transform (commonly known as Haar transform) on the spatial domain image. Specifically, a one-dimensional discrete wavelet transform can be performed on the row pixels of the spatial domain image to obtain a first transformed image; another one-dimensional discrete wavelet transform can be applied to the column pixels of the first transformed image to calculate the wavelet coefficients; four second transformed images, each one-quarter the size of the original spatial domain image, are obtained. These four second transformed images represent the low-frequency information, horizontal high-frequency information, vertical high-frequency information, and diagonal high-frequency information of the first transformed image, respectively; the second transformed images are then interpolated and upsampled to obtain a frequency domain image with the same size as the original spatial domain image.

[0070] The spatial domain encoder 10, frequency domain encoder 20, and decoder 30 can adopt a convolutional neural network (CNN) model or a Swing Transformer model (the first part of the name Swing Transformer, Swing, comes from Shifted Windows, which is also a key feature of Swing Transformer). However, the CNN model only has local perception capabilities and cannot establish long-range dependencies and global context connections. Defect detection requires a combination of global and local information for greater accuracy. Through extensive experiments, the inventors discovered that the Swing Transformer model, by adding an inter-window attention mechanism (see existing Swing Transformer model introductions for details, which will not be repeated here), can enhance the extraction of global semantic features, i.e., it can extract fiber optic defect features with long-range dependencies and global context connections, thereby improving defect detection capabilities. Therefore, preferably, in this embodiment, the spatial domain encoder 10, frequency domain encoder 20, and decoder 30 adopt the Swing Transformer model. The spatial domain encoder 10 and frequency domain encoder 20 have identical structures, including at least an image block layer 100 and several cascaded depth feature map output modules 200, wherein:

[0071] The image segmentation layer 100 is used to divide the input spatial domain image or frequency domain image into several blocks of the same size.

[0072] The depth feature map output module 200 includes a first depth feature map output module 201 and several second depth feature map output modules 202, wherein:

[0073] Combination Figure 3 As shown, the first deep feature map output module 201 includes a linear embedding layer 2011 and a SwingTransformer block layer; the second deep feature map output module 202 includes a block merging layer 2021 and a SwingTransformer block layer; wherein:

[0074] The linear embedding layer 2011 is used to perform a linear transformation on the block map output by the image block layer 100, so that the feature map structure size output by the spatial domain encoder 10 and the frequency domain encoder 20 in the linear embedding layer 2011 is consistent.

[0075] Linear embedding layers (2011) are essentially linear transformations; their function is to perform a dimensionality reduction transformation. For example, after a spatial domain image (shape 3×H×W) is divided into blocks, the feature map size is... The frequency domain image (4×H×W) is after being divided into blocks. Although they have different shapes, they are all transformed into feature maps of the same size with different linear dimensions.

[0076] The block merging layer 2021 is used to reduce the resolution and increase the dimensionality of the input feature map. The block merging layer 2021 increases the dimensionality of the features (higher dimensions contain more effective features) while reducing the size of the feature map (reducing redundant information), that is, it reduces the feature map resolution while increasing the feature dimensionality. The aim is to enhance the representational ability of extracted features.

[0077] The Swing Transformer block layer is used to extract features from the input feature map according to preset rules. Generally, it extracts fiber optic defect features with long-range dependencies and global context connectivity to obtain a feature map of the same size as the input.

[0078] like Figure 2 As shown, the spatial domain encoder 10 includes a first depth feature map output module 201 and three second depth feature map output modules 202. The spatial domain feature generated by the first depth feature map output module 201 is E1. The spatial domain features generated by the three second-depth feature map output modules 202 are E2, respectively. Among them, D, 2D, 4D, and 8D represent doubling the dimension of the feature map.

[0079] Specifically, the spatial domain encoder 10 receives a spatial domain image (3×H×W in shape), passes it through an image block layer 100, and divides the image into 4×4=16 blocks, resulting in a total of 3×16=48 image feature maps (3×H×W in shape). ), and then converted to a size of by the linear embedding layer 2011 of the first deep feature map output module 201. The feature map is then processed by the Swing Transformer block layer of the first deep feature map output module 201 for representation learning, resulting in a size of [missing value]. The spatial domain depth feature map E1. Then, the block merging layer 2021 of the first and second depth feature map output modules 202 performs a 2x downsampling on feature map E1, reducing the feature map resolution while increasing the feature dimension, resulting in a size of... The feature map is then passed through a Swim Transformer block layer to obtain a size of [missing information]. The spatial domain depth feature map E2. Then, after passing through the second second depth feature map output module 202, a size of... The spatial domain depth feature map E3. Finally, after passing through the third second depth feature map output module 202, a size of [size missing] is obtained. Spatial domain depth feature map E4.

[0080] The frequency domain encoder 20 includes a first depth feature map output module 201 and three second depth feature map output modules 202. The spatial domain feature generated by the first depth feature map output module 201 is F1. C is the number of channels), and the spatial domain features generated by the three second-depth feature map output modules 202 are F2 and F2, respectively.

[0081] Specifically, the frequency domain encoder 20 receives a frequency domain image (4×H×W in shape), passes it through an image block layer 100, and divides the image into 4×4=16 blocks, resulting in a total of 4×16=64 image feature maps (in shape...). ), and then converted to a size of by the linear embedding layer 2011 of the first deep feature map output module 201. The feature map is then processed by the Swing Transformer block layer of the first deep feature map output module 201 for representation learning, resulting in a size of [missing value]. The frequency domain depth feature map F1 is then processed. Next, the block merging layer 2021 of the first and second depth feature map output modules 202 downsamples feature map F1 by a factor of 2, reducing the feature map resolution while increasing the feature dimension, resulting in a map of size [size missing]. The feature map is then passed through a Swim Transformer block layer to obtain a size of [missing information]. The frequency domain depth feature map F2. Then, after passing through the second second depth feature map output module 202, a size of... The frequency domain depth feature map F3 is then processed by the third second depth feature map output module 202 to obtain a value of [size missing]. Frequency domain depth feature map F4. It should be noted that because the spatial domain image input to spatial domain encoder 10 has a different structure than the frequency domain image input to frequency domain encoder 20, the linear embedding layers 2011 of spatial domain encoder 10 and frequency domain encoder 20 differ, but the final output is a feature map of the same size.

[0082] Preferably, the decoder 30 includes a plurality of cascaded depth feature map decoding modules 300 and a linear projection layer 400. In this embodiment, it includes four depth feature map decoding modules 300 and one linear projection layer 400, wherein:

[0083] The deep feature map decoding module includes a fusion layer, a Swing Transformer block layer, and a block expansion layer, wherein:

[0084] The fusion layer is used to concatenate the input feature maps according to channels and positions, and then output them to the Swing Transformer block layer after linear transformation. The input feature maps include spatial domain feature maps, frequency domain feature maps, and feature maps output by the feature map decoding module of the previous depth. The Swing Transformer block layer is used to extract features from the input feature maps according to preset rules and then output them to the block expansion layer. The block expansion layer is used to increase the resolution and reduce the dimensionality of the input feature maps before outputting them.

[0085] Combination Figure 4 As shown, the deep feature map decoding module 300 includes a fusion layer 3001, a SwingTransformer block layer, and a block expansion layer 3002, wherein:

[0086] The fusion layer 3001 is used to concatenate the input feature maps according to channels and positions, and then output them to the Swing Transformer block layer after linear transformation. The input feature maps include spatial domain feature maps, frequency domain feature maps, and feature maps output by the previous depth feature map decoding module 300, all located at the same depth. It can be understood that the first fusion layer only fuses the spatial domain depth feature map E4 and the frequency domain depth feature map F4, while the second fusion layer fuses the spatial domain depth feature map E3, the frequency domain depth feature map F3, and the output feature map of the previous depth feature map decoding module 300.

[0087] The Swing Transformer block layer is used to extract features from the image input to it according to preset rules.

[0088] The block expansion layer 3002 is used to increase the resolution and reduce the dimensionality of the input feature map before outputting it. The function of the block expansion layer 3002 is to perform a dimensionality and scale transformation, reshaping the high-dimensional feature map into a higher-resolution feature map (2x upsampling), and correspondingly reducing the feature dimension to half of the original dimension. Inside the block expansion layer 3002, a linear transformation layer first increases the feature dimension to twice its original value. Then, the features are rearranged, increasing both the length and width of the feature map to twice their original value, at which point the feature dimension is automatically reduced by a factor of four.

[0089] Specifically, the spatial domain depth feature map E4 and the frequency domain depth feature map F4 are used as inputs to the decoder, and are fused and stitched together in the first fusion layer 3001 to form a matrix of size [size missing]. The feature map, after undergoing dimensional and resolution transformation through the Swing Transformer block layer and block expansion layer 3002 of the first deep feature map decoding module 300, results in a size with doubled resolution. The decoded feature map D1 is then obtained. Next, the spatial domain depth feature map E3, the frequency domain depth feature map F3, and the decoded feature map D1 are concatenated and fused together, and input into the Swing Transformer block layer of the second depth feature map decoding module 300, resulting in a size of... The feature blocks, after passing through the block expansion layer 3002 of the second depth feature map decoding module 300, yield a size of... The decoded feature block D2. Spatial domain depth feature map E3, frequency domain depth feature map F3, and decoded feature map D2 are concatenated and fused, then input into the SwimTransformer block layer of the third depth feature map decoding module 300, resulting in a block of size D2. The feature blocks, after being processed by the block expansion layer 3002 of the third depth feature map decoding module 300, yield a size of The decoding feature block D3. Spatial domain depth feature map E1, frequency domain depth feature map F1, and decoding feature map D3 are concatenated and fused, then input together into the Swim Transformer block layer of the third depth feature map decoding module 300, resulting in a block of size D3. The feature blocks, after passing through the block expansion layer 3002, yield a depth feature block D4 of size (48×H×W). Understandably, in practical applications, the parameters within the block expansion layer 3002 are obtained through training with training samples.

[0090] The linear projection layer 400 maps the feature map output by a connected depth feature map decoding module 300 to a feature map with the same size as the original spatial domain image and the number of channels equal to the number of defect categories. For example, it transforms the output 48×H×W to C×H×W. Finally, the decoder 30 inputs the feature map output by the linear projection layer 400 into the segmentation result image generation module 40.

[0091] Example 3

[0092] The present invention also discloses a method for optical fiber end-face inspection, which utilizes the optical fiber end-face inspection device as described in Embodiment 1 or 2, and includes at least the following steps:

[0093] S101, the spatial domain encoder 10 receives the input spatial domain image of the end face of the optical fiber to be detected, and obtains several spatial domain feature maps of different sizes, each spatial domain feature map corresponding to a different depth of the spatial domain encoder 10.

[0094] Specifically, downsampling of spatial domain images means reducing the resolution and increasing the dimensionality of the resulting spatial domain feature map, thereby enabling the abstract representation of the input image using high-dimensional information. The higher the dimensionality, the more effective features it contains. At the same time, it reduces the size of the feature map and reduces redundant information.

[0095] S102, the frequency domain encoder 20 receives the frequency domain image of the end face of the optical fiber to be detected and obtains several frequency domain feature maps of different sizes, each frequency domain feature map corresponding to a different depth of the frequency domain encoder 20.

[0096] The frequency domain encoder 20 downsamples the frequency domain image, which means that the resolution of the frequency domain feature map is reduced and the dimension is increased. This allows the input image to be abstractly represented using high-dimensional information. The higher the dimension, the more effective features it contains. At the same time, the size of the feature map is reduced, and redundant information is reduced.

[0097] S103, the decoder 30 performs feature fusion, feature extraction and upsampling on the embedded spatial domain feature maps and frequency domain feature maps at different depths, and outputs pixel-level classification result feature maps.

[0098] Specifically, feature fusion, feature extraction, and upsampling (e.g., doubling the upsampling each time) are performed on the embedded spatial domain feature maps and frequency domain feature maps at different depths. Then, the feature map output by the last upsampling is mapped by linear projection to a feature map with the same size as the original spatial domain image and the number of channels equal to the number of defect categories, outputting a pixel-level classification result feature map C×H×W.

[0099] S104: Based on the feature map of the pixel-level classification results, obtain the probability score of each pixel being predicted as a certain defect. Take the defect category with the highest probability score for each pixel as the classification result of that pixel. Perform pixel segmentation on the defect based on the classification results of all pixels to obtain the location and size of the fiber end face defect.

[0100] Specifically, based on the pixel-level classification result feature map, the probability score of each pixel being predicted as a certain type of defect is obtained. The defect category with the highest probability score for each pixel is taken as the classification result for that pixel. That is, the fiber optic end-face inspection device predefines defect categories (such as scratches, specks, dirt, etc.) and defect severity (e.g., for scratches, obvious scratches, minor scratches) during the training phase. Then, defect segmentation is performed based on the classification results of all pixels, that is, the region defined as a defect is extracted. The location of this region is the defect location, and the number of pixels within this region is the defect size, which can be represented by the perimeter or area of ​​the defect. Obtaining the location and size of the defect enables defect detection of the fiber optic end face. In practical applications, depending on different customer needs, defects need to be classified. Some small defects are acceptable (such as small specks on the edge of the product), while large defects (such as large dirt in the center of the fiber optic end face) are considered non-compliant.

[0101] This invention focuses on image defect detection at the pixel level, combining the advantages of both spatial and frequency domain images (i.e., spatial domain images retain the characteristics of the original image, while frequency domain images provide texture information). This allows for the detection of both obvious and subtle defects. Furthermore, the shallower the spatial and frequency domain feature maps (10 and 20), the lower their dimensionality, yet they retain more pixel detail. Therefore, the decoder (30) performs stitching and decoding of spatial and frequency domain feature maps at different depths, resulting in less feature information loss compared to decoding only the deepest feature map. This ensures higher accuracy in fiber optic end-face detection.

[0102] Example 4

[0103] Since the spatial domain image and frequency domain image of the fiber end face to be inspected cannot be directly obtained from the fiber end face, in some embodiments, the fiber end face inspection method includes the following steps:

[0104] S201, the region of interest is extracted from the image of the fiber end face to be inspected according to preset rules to obtain a spatial domain image.

[0105] Specifically, the spatial domain image is a three-dimensional image, which can be represented by 3×H×W, where 3 represents the dimension, H represents the height of the image, and W represents the width of the image. In some embodiments, the spatial domain image can be obtained by extracting regions of interest (ROIs) from the fiber end face image to be detected according to preset rules. The preset rules include selecting or using methods such as setting thresholding on the spatial domain image to extract irregular shapes including points, lines, and surfaces. In this embodiment, the spatial domain image can be obtained by extracting regions of interest from the fiber end face image through threshold binarization.

[0106] S202, perform Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

[0107] Specifically, the frequency domain image is a four-dimensional image, which can be represented by 4×H×W, where 4 represents the dimension, H represents the height of the image, and W represents the width of the image. In some embodiments, the frequency domain image generation module can obtain it by performing a Haar wavelet transform (commonly known as Haar transform) on the spatial domain image. Specifically, a one-dimensional discrete wavelet transform can be performed on the row pixels of the spatial domain image to obtain a first transformed image; another one-dimensional discrete wavelet transform can be applied to the column pixels of the first transformed image to calculate the wavelet coefficients; four second transformed images, each one-quarter the size of the original spatial domain image, are obtained. These four second transformed images represent the low-frequency information, horizontal high-frequency information, vertical high-frequency information, and diagonal high-frequency information of the first transformed image, respectively; the second transformed images are then interpolated and upsampled to obtain a frequency domain image with the same size as the original spatial domain image.

[0108] S203, the spatial domain encoder 10 receives the input spatial domain image of the end face of the optical fiber to be detected, and obtains several spatial domain feature maps of different sizes, each spatial domain feature map corresponding to a different depth of the spatial domain encoder 10.

[0109] S204, the frequency domain encoder 20 receives the frequency domain image of the end face of the optical fiber to be detected and obtains several frequency domain feature maps of different sizes, each frequency domain feature map corresponding to a different depth of the frequency domain encoder 20.

[0110] S205, the decoder 30 performs feature fusion, feature extraction and upsampling on the embedded spatial domain feature maps and frequency domain feature maps at different depths, and outputs pixel-level classification result feature maps.

[0111] S206. Based on the feature map of the pixel-level classification results, obtain the probability score of each pixel being predicted as a certain defect. Take the defect category with the highest probability score for each pixel as the classification result of that pixel. Perform pixel segmentation on the defect based on the classification results of all pixels to obtain the location and size of the fiber end face defect.

[0112] Example 5

[0113] This invention also discloses a fiber optic end-face detection model training method for training fiber optic end-face detection models as shown in Embodiment 1 or 2, aiming to improve the accuracy of the fiber optic end-face detection model. The training method includes:

[0114] The sample training set is fed into the fiber optic end-face detection model, and the model is optimized using a loss function. The sample training set includes several spatial domain images, frequency domain images, and defect category label images corresponding to each spatial domain image and frequency domain image. The defect category label image can be represented by a single value or can be the same size as the feature map output by the linear projection layer 400 (C×H×W). Each pixel has one label (different values ​​represent different defect categories) or C labels (if it belongs to a certain category of defect, that category is 1; otherwise, it is 0). For example, in some embodiments, the fiber optic end-face detection model divides defect categories into four categories (no defect, scratch, noise, dirt) during the training phase, i.e., C=4, then each pixel in H×W has four channels.

[0115] The loss function considers both the individuality and overall nature of pixel classification to jointly constrain the penalty strength. The individuality loss of pixel classification measures the distance between the predicted value and the pixel label value. Assuming g... c y represents the true label (0 or 1) of a pixel belonging to category c. c This represents the predicted probability (within the range of 0 to 1) that the pixel belongs to class c, which can be determined based on the predicted value y. c The weight of individual loss is adjusted accordingly; that is, the more accurate the prediction, the smaller the weight, and the worse the prediction, the larger the weight (the greater the penalty).

[0116] When constructing the loss function, set the base term to the cross-entropy function -g. c log(y c Simultaneously, the individuality loss weight of pixels (1-y) is dynamically adjusted based on the prediction results of the samples. c ) 2 Then, the individual loss of all pixels is summed and averaged to obtain formula (1):

[0117]

[0118] Where C represents the category label, P represents the number of pixels in the image, and g c y represents the true label (0 or 1) of a pixel belonging to category c. c This represents the predicted probability (in the range of 0 to 1) that the pixel belongs to class c.

[0119] Additionally, the overall loss for pixel classification is used to measure the distance between two sets. Each category has a ground truth label map and a predicted probability map. To calculate the loss between the ground truth label map A and the predicted probability map B for each category, the overlap between the sets can be used. This is used to measure the similarity (ranging from 0 to 1) between two sets, and then to measure their distance. (Range 0-1), this distance is used to calculate the loss. In the loss function, g...c For example, the label of a pixel in set A, y c The predicted probability of a pixel belongs to set B in the formula, and this is then accumulated pixel by pixel to obtain the final calculation. The losses of the predicted and labeled images for each category are then weighted and accumulated to obtain the overall loss. Simultaneously, the overall category loss weights ω for different categories are calculated based on the frequency of occurrence of different defect categories. c This step yields formula (2):

[0120]

[0121] Where ε is a constant to avoid the denominator being 0. Where f k This represents the frequency of category label k in the training samples.

[0122] Finally, by adding formulas (1) and (2), the final loss function is obtained, expressed by formula (3). This solves the problems of poor classification results caused by small targets, difficult sample learning, and class imbalance (individuality loss weight (1-y)). c ) 2 It can solve the problem of difficult learning from samples, and the holistic loss can solve the problem of small objectives. The holistic class loss has weights ω. c (To address the class imbalance problem). The specific calculation of the loss function is as follows:

[0123]

[0124] Where C represents the category label, P represents the number of pixels in the image, and g c y represents the true label (0 or 1) of a pixel belonging to category c. c This represents the predicted probability (in the range of 0 to 1) that the pixel belongs to class c. ε is a constant to avoid the denominator being 0. Where f k This represents the frequency of category label k in the training samples.

[0125] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0126] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term “comprising” as used in the specification or claims is interpreted in a manner similar to the term “including,” just as “including,” is interpreted as a conjunction in the claims. Additionally, the use of any term “or” in the specification of the claims is intended to mean “non-exclusive or.”

Claims

1. A fiber optic end-face inspection device, characterized in that, It includes a spatial domain encoder, a frequency domain encoder, a decoder, and a segmentation result map generation module, wherein: The spatial domain encoder is used to extract features and downsample the spatial domain image of the input optical fiber end face to obtain several spatial domain feature maps of different sizes, and each spatial domain feature map corresponds to a different depth of the spatial domain encoder. The frequency domain encoder is used to extract features and downsample the frequency domain image of the input optical fiber end face to obtain several frequency domain feature maps of different sizes, and each frequency domain feature map corresponds to a different depth of the frequency domain encoder. The decoder is used to perform feature fusion, feature extraction and upsampling on the embedded spatial domain feature map and frequency domain feature map at different depths, and output pixel-level classification result feature map; The segmentation result image generation module is used to obtain the probability score of each pixel being predicted as a certain defect based on the pixel-level classification result feature map. The defect category with the highest probability score for each pixel is taken as the classification result of that pixel. The defect is segmented into pixels based on the classification results of all pixels to obtain the location and size of the fiber end face defect. The spatial domain encoder and frequency domain encoder have the same structure, adopting a Swing Transformer model structure, and include at least one image segmentation layer and several cascaded depth feature map output modules. The image segmentation layer is used to divide the input spatial domain image or frequency domain image into several blocks of the same size. The depth feature map output modules include a first depth feature map output module and several second depth feature map output modules, wherein: the first depth feature map output module includes a linear embedding layer and a Swing Transformer block layer; the second depth feature map output modules include a block merging layer and a Swing Transformer block layer; the linear embedding layer is used to perform a linear transformation on the block map output by the image segmentation layer, so that the feature map structure size output by the spatial domain encoder and the frequency domain encoder in the linear embedding layer is consistent; the block merging layer is used to reduce the resolution and increase the dimension of the feature map input to it; the Swing Transformer block layer is used to extract features from the feature map input to it according to preset rules. The decoder comprises several cascaded depth feature map decoding modules and a linear projection layer. Each depth feature map decoding module includes a fusion layer, a Swing Transformer block layer, and a block expansion layer. The fusion layer concatenates the input feature maps by channel and position, performs a linear transformation, and outputs the result to the Swing Transformer block layer. The input feature maps include spatial domain feature maps, frequency domain feature maps, and feature maps output by the previous depth feature map decoding module. The Swing Transformer block layer extracts features from the input feature maps according to preset rules and outputs the result to the block expansion layer. The block expansion layer increases the resolution and reduces the dimensionality of the input feature maps before outputting them. The linear projection layer maps the feature map output by a connected depth feature map decoding module to a feature map with the same size as the original spatial domain image and a channel number equal to the number of defect categories.

2. The optical fiber end-face inspection device as described in claim 1, characterized in that, It also includes a spatial domain image generation module, which is used to extract the region of interest from the image of the fiber end face to be inspected according to preset rules to obtain a spatial domain image.

3. The optical fiber end-face inspection device as described in claim 1, characterized in that, The fiber optic end face detection device also includes a frequency domain image generation module, which performs Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

4. The optical fiber end-face inspection device as described in claim 1, characterized in that, The spatial domain encoder, frequency domain encoder, and decoder adopt a neural network convolutional CNN model or a Swing Transformer model structure.

5. The optical fiber end-face inspection device as described in claim 1, characterized in that, The segmentation result image generation module is pre-set with a normalized exponential function, softmax, to obtain the probability score of each pixel being predicted as a certain defect. The content of the normalized exponential function, softmax, is as follows: in Let C be the output value of a pixel in the i-th channel, and C be the number of categories.

6. A method for inspecting the end face of an optical fiber, applicable to the optical fiber end face inspection device as described in any one of claims 1 to 5, characterized in that, At least the following steps are included: The spatial domain image of the fiber end face to be detected is received, and several spatial domain feature maps of different sizes are obtained. Each spatial domain feature map corresponds to a different depth of the spatial domain encoder. The frequency domain image of the fiber end face to be tested is received, and several frequency domain feature maps of different sizes are obtained. Each frequency domain feature map corresponds to a different depth of the frequency domain encoder. The embedded spatial domain feature maps and frequency domain feature maps at different depths are fused, extracted and upsampled to output pixel-level classification result feature maps. Based on the feature map of the pixel-level classification results, the probability score of each pixel being predicted as a certain defect is obtained. The defect category with the highest probability score for each pixel is taken as the classification result of that pixel. Based on the classification results of all pixels, the defect is segmented into pixels to obtain the location and size of the fiber end face defect.

7. The fiber optic end-face inspection method as described in claim 6, characterized in that, It also includes the following steps: The region of interest is extracted from the image of the fiber end face to be inspected according to preset rules to obtain a spatial domain image; Perform Haar wavelet transform on the row pixels of the obtained spatial domain image to obtain a frequency domain image with the same length and width as the spatial domain image.

8. A method for training a fiber optic end-face detection model, used to train the fiber optic end-face detection device as described in any one of claims 1 to 5, characterized in that, The training method includes: feeding a sample training set into an optical fiber end-face detection device, optimizing the optical fiber end-face detection device using a loss function, wherein the sample training set includes several spatial domain images, frequency domain images, and defect category label images corresponding to each spatial domain image and frequency domain image, and the loss function is: Where C represents the category label, and P represents the number of pixels in the image. The true label representing the pixel belonging to category c. This represents the predicted probability that a pixel belongs to class c. To avoid the denominator being 0, a constant is used. in This represents the frequency of category label k in the training samples.

Images