X-ray image foreign object detection method based on self-supervised learning

Abstract

The application provides an X-ray image foreign matter detection method based on self-supervised learning. The method comprises the following steps: step 1: constructing a ConvMAE model based on structure reparameterization, denoted as a RepConvMAE self-supervised model, comprising an Encoder for extracting a multi-scale feature map of an input image and a Decoder for reconstructing the image based on the extracted multi-scale feature map; step 2: pre-training the RepConvMAE self-supervised model by using an X-ray image dataset; step 3: constructing a detection model EFFR-CNN, which comprises, from a shallow layer to a deep layer, a backbone network, an FPN layer, a shared feature layer, an RPN layer, an ROI Pooling layer and a full connection layer; the backbone network adopts the Encoder in the pre-trained RepConvMAE self-supervised model; step 4: training the detection model EFFR-CNN by using an X-ray foreign matter image dataset; and step 5: inputting an X-ray image to be detected into the trained detection model EFFR-CNN to obtain a detection result.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method for detecting foreign objects in X-ray images based on self-supervised learning. Background Technology

[0002] Needle detectors are primarily used to detect broken needles and metal fragments in textiles, preventing these sharp metal foreign objects from injuring consumers. Traditional needle detectors operate on the principle of magnetic induction, using a permanent magnet inside the probe. This allows them to detect only ferromagnetic metals (iron, cobalt, nickel, and alloys containing these metals as impurities), but not other high-purity, fine metals. Newer needle detectors utilize deep learning-based foreign object detection algorithms, solving this problem.

[0003] However, current foreign object detection algorithms have certain shortcomings. For example, patent document CN112525931A discloses a metal foreign object recognition network, Super-FODNet algorithm. This algorithm requires collecting tens of thousands of X-ray images of shoes and clothing with and without metal foreign objects, and requires a large amount of manpower to annotate the images. The annotation cost is high and consumes a lot of manpower and resources. Moreover, manual annotation is not entirely accurate, and omissions and errors may occur when annotating data, which will affect the training speed and accuracy. Patent document CN109886935A discloses a pavement foreign object detection method based on deep learning. It uses deep learning to judge and identify multiple foreign object blocks and output foreign object categories. This method requires an equal number of samples for each type, but it is difficult to collect an equal number of samples. Summary of the Invention

[0004] To address the issues of reduced detection accuracy caused by insufficient and imbalanced data samples, as well as the inaccuracy of manually labeled data, this invention provides a self-supervised learning-based X-ray image foreign object detection method. This method can solve the problems of reduced detection accuracy caused by insufficient and imbalanced data samples, as well as the inaccuracy of manually labeled data; it can also improve the detection performance of the detection model for small objects (such as broken metal needles).

[0005] This invention provides a method for foreign object detection in X-ray images based on self-supervised learning, comprising:

[0006] Step 1: Construct a ConvMAE model based on structural reparameterization, denoted as RepConvMAE self-supervised model, which includes an Encoder for extracting multi-scale feature maps of the input image and a Decoder for reconstructing the image based on the extracted multi-scale feature maps;

[0007] Step 2: Pre-train the RepConvMAE self-supervised model using an X-ray image dataset;

[0008] Step 3: Construct the detection model EFFR-CNN, which consists of the following layers from shallow to deep: backbone network, FPN layer, shared feature layer, RPN layer, ROI Pooling layer, and fully connected layer; the backbone network uses the Encoder from the pre-trained RepConvMAE self-supervised model.

[0009] Step 4: Train the detection model EFFR-CNN using the X-ray foreign object image dataset;

[0010] Step 5: Input the X-ray image to be tested into the trained detection model EFFR-CNN to obtain the detection results.

[0011] Furthermore, the Encoder in the RepConvMAE self-supervised model consists of four stages from shallow to deep; the first three stages have the same structure, each including a Patch Embedding layer and a MaskedConvolution Block based on structural reparameterization; the last stage includes a Patch Embedding layer and a Transformer Block.

[0012] The Masked Convolution Block based on structure reparameterization consists of the following layers from shallow to deep: Batch Norm layer, 1*1 convolutional layer, K*K convolutional layer based on structure reparameterization, Batch Norm layer, 1*1 convolutional layer, and 1*1 convolutional layer. The K*K convolutional layer based on structure reparameterization refers to the following: during training, the K*K convolutional layer is replaced with three branches: a 1*K convolutional layer followed by a Batch Norm layer, a K*1 convolutional layer followed by a Batch Norm layer, and a K*K convolutional layer followed by a Batch Norm layer. During inference, these three branches are replaced with a single K*K convolutional layer.

[0013] Furthermore, the masking strategy of the RepConvMAE self-supervised model is as follows: the input of the last stage is directly randomly masked, and then the mask of the last stage is upsampled by different multiples to obtain the masks of the first three stages; wherein, the mask of each patch is represented by 0 or 1, where 0 represents masking and 1 represents no masking.

[0014] Furthermore, the Decoder in the RepConvMAE self-supervised model includes a Linear layer and a Transformer Block; wherein, the multi-scale feature maps are added after strided convolution and then fed into the Linear layer, and the output of the Linear layer is padded with masked patches and then fed into the Transformer Block.

[0015] Furthermore, the processing procedure of the Linear layer is represented by formula (2):

[0016] E d =Linear(StrideConv(E1,8)+StrideConv(E2,4)+StrideConv(E3,2)+E4) (2)

[0017] Among them, E d The output of the Linear layer is represented by StrideConv(,X), which represents a convolution with a stride of X. E1, E2, E3, and E4 represent feature maps of four different scales arranged from largest to smallest.

[0018] Furthermore, the FPN adopts an improved FPN; the improved FPN refers to adding a fusion factor to the original FPN, specifically including: the improved FPN uses formula (3) to aggregate adjacent feature layers;

[0019]

[0020] in, This indicates that a 1x1 convolutional layer achieves channel matching, F UP This indicates a doubling upsampling operation to achieve feature map size matching. Conv() represents the convolution operation used for feature processing, α is the fusion factor, and p i This represents the i-th feature layer.

[0021] Furthermore, the value of the fusion factor α is determined using formula (4):

[0022]

[0023] in, and P i+1 Layers and P i The number of objects on the layer.

[0024] The beneficial effects of this invention are:

[0025] (1) The X-ray image foreign object detection method based on self-supervised learning disclosed in this invention includes a pre-trained model and a detection model. The pre-trained model is trained in a self-supervised manner, which does not require a large amount of labeled datasets, thus saving manpower costs. It also does not require the number of samples of each type in the data to be balanced, thereby solving the problem of the impact of data imbalance on the model performance. In the detection model, in addition to using the traditional FPN structure to achieve feature fusion, an improved FPN layer is proposed to further improve the detection accuracy of the model for small objects.

[0026] (2) The pre-trained model adopts a multi-level encoder, which can extract multi-scale features and improve the expressive power of the feature map; it also adds structural reparameterization operation to the pre-trained model, so that the pre-trained model has a large number of parameters during training while ensuring the speed of the model during inference, thus improving the detection efficiency of the model in practical applications.

[0027] (3) In the detection model, the Encoder in the pre-trained model is used as the backbone network to fuse multi-scale features. Combined with the improved FPN structure, the detection accuracy of the model for small objects is further improved, which can meet the requirements of accuracy and efficiency in actual industry.

[0028] (4) This invention has a wide range of applications and can be applied to foreign object detection in industrial production fields such as food, textiles, medicine, and electronics. Attached Figure Description

[0029] Figure 1 This is a flowchart of the X-ray image foreign object detection method based on self-supervised learning provided in an embodiment of the present invention;

[0030] Figure 2 This is a network structure diagram of the RepConvMAE self-supervised model provided in this embodiment of the invention;

[0031] Figure 3 This is a structural diagram of the Masked Convolution Block based on structural reparameterization provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the structure reparameterization operation in the Masked Convolution Block based on structure reparameterization provided in an embodiment of the present invention;

[0033] Figure 5 This is a network structure diagram of the EFFR-CNN detection algorithm provided in this embodiment of the invention; Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment of the invention provides a method for detecting foreign objects in X-ray images based on self-supervised learning, including the following steps:

[0037] S101: Construct a ConvMAE model based on structural reparameterization, denoted as RepConvMAE self-supervised model, which includes an Encoder for extracting multi-scale feature maps of the input image and a Decoder for reconstructing the image based on the extracted multi-scale feature maps; RepConvMAE: Reparam Convolution Masked AutoEncoders.

[0038] Specifically, by incorporating structural reparameterization into the model, the model can acquire a large number of parameters during training while maintaining its inference speed, thus improving its detection efficiency in practical applications. Furthermore, compared to MAE models that can only extract single-scale features, the RepConvMAE self-supervised model in this embodiment can extract multi-scale features.

[0039] S102: The RepConvMAE self-supervised model is pre-trained using an X-ray image dataset;

[0040] Specifically, the model is pre-trained using an open-source X-ray image dataset. The dataset images are divided into several patches (e.g., 16*16) and fed into the RepConvMAE self-supervised model.

[0041] S103: Construct the detection model EFFR-CNN, such as Figure 5 As shown, from shallow to deep layers, the structure includes: backbone network, FPN layer, shared feature layer, RPN layer, ROI Pooling layer, and fully connected layer; the backbone network uses the pre-trained Encoder from the RepConvMAE self-supervised model; EFFR-CNN: Effective Fusion Factor Region-CNN;

[0042] Specifically, the backbone network first extracts features, then the FPN layer fuses the extracted features to obtain a shared feature layer. This shared feature layer enters the RPN (Region Prospect Network) layer to obtain candidate boxes. The ROI Pooling layer combines with the shared feature layer to filter all candidate boxes. The filtered candidate boxes are then processed by a fully connected layer and softmax to generate the location information, confidence score, and foreign object category of the detection boxes. The fully connected layer processes the following: the IOU (Intersection over Union) between the candidate box and the ground truth box is compared with an initial threshold to obtain the four-dimensional information of the foreign object detection boxes that meet the conditions. This comparison process is repeated continuously, with the output value of the previous comparison process used as the input value of the current comparison process, and the IOU threshold of the current comparison process is continuously increased. Finally, the detection boxes that meet the conditions are obtained, and the location information (e.g., four-dimensional information), confidence score (e.g., IoU value), and foreign object category of the detection boxes are output.

[0043] S104: The detection model EFFR-CNN is trained using an X-ray foreign object image dataset;

[0044] Specifically, X-ray images of the objects to be detected are collected as the dataset for the detection model. Multiple batches of collected objects are labeled, and the labeling information includes bounding boxes (x,y,w,h) and the category label of each bounding box. Then, the dataset is divided into training set and test set in a 4:1 ratio.

[0045] S105: Input the X-ray image to be tested into the trained detection model EFFR-CNN to obtain the detection result.

[0046] Furthermore, in practical applications, before inputting the dataset into the detection model for training or testing, preprocessing is required to augment the dataset, thereby improving detection accuracy and preventing overfitting. This preprocessing includes at least one of the following: image mixing, label smoothing, random geometric transformation, random angle rotation, and random color transformation. Random geometric transformations (all with defined ranges and probabilities) include: random cropping, random expansion, random horizontal flipping, and random stretching (random interpolation); random color transformations include: random transformations of contrast, brightness, saturation, and chroma.

[0047] In the detection model EFFR-CNN of this embodiment, the backbone network used for feature extraction (i.e., the Encoder in the RepConvMAE self-supervised model) is pre-trained using a self-supervised method. Compared with supervised pre-training, the training method of this invention can complete the pre-training process with only a small amount of labeled X-ray image data in the downstream task, thereby saving a lot of manpower for data labeling.

[0048] Example 2

[0049] The inventors believe that convolutional neural networks can reduce local redundancy and computational cost through small-scale convolutions, but they are limited in capturing global dependencies. ViT (Vision Transformer) can effectively capture long-range dependencies through self-attention, but it struggles to effectively encode local features in shallow networks. Therefore, this embodiment combines ViT with convolutional neural networks to improve the model's performance in detecting metallic foreign objects such as broken needles.

[0050] like Figure 2 As shown, as one possible implementation, the Encoder in the RepConvMAE self-supervised model adopts a multi-level structure, consisting of four stages (stage1, stage2, stage3, and stage4) from shallow to deep. The first three stages have the same structure, each including a Patch Embedding layer and a Masked Convolution Block (also known as Masked Re-Param Convolution Block, or Masked RepConv Block for short) based on structural reparameterization. The last stage includes a Patch Embedding layer and a Transformer Block. Each stage outputs a feature map of one scale.

[0051] like Figure 3 As shown, the Masked Convolution Block based on structure reparameterization includes, from shallow to deep, a Batch Norm layer, a 1*1 convolutional layer, a K*K convolutional layer based on structure reparameterization, a Batch Norm layer, a 1*1 convolutional layer, and a 1*1 convolutional layer; wherein, as... Figure 4 As shown, the K*K convolutional layer based on structural reparameterization refers to the following: during training, the K*K convolutional layer is replaced with three branches: a 1*K convolutional layer followed by a Batch Norm layer, a K*1 convolutional layer followed by a Batch Norm layer, and a K*K convolutional layer followed by a Batch Norm layer; during inference, the three branches are replaced with a single K*K convolutional layer. Structural reparameterization can be expressed by formula (1):

[0052] Reparam(K*K)=BN(Conv(K,K))+BN(Conv(1,K))+BN(Conv(K,1)) (1)

[0053] Where Reparam(K*K) represents the reparameterized convolution, Conv(K,K) represents the K*K convolution, and BN() represents the Batch Norm layer. Preferably, in this embodiment, K=4.

[0054] In addition, it should be noted that global self-attention has a large receptive field, but the computational cost is too high. Therefore, in practical applications, the following settings can be adopted for the 6 layers of stage 4 in RepConvMAE as needed: {1, 3, 5} use 7*7 window size Swing Attention, and {2, 4, 6} use global self-attention to reduce the amount of computation and the use of GPU.

[0055] In this embodiment, the Encoder adopts a multi-level structure, which is more complex than a single-level structure. This structure may require longer detection time during actual foreign object detection. To overcome this problem and reduce the time spent detecting X-ray images, this embodiment employs structural reparameterization. Structural reparameterization refers to constructing a series of parameters during training and then equivalently converting these parameters into another set of parameters during inference. More parameters during training generally result in better model performance, while fewer parameters lead to faster inference. Structural reparameterization effectively combines these two advantages. In this embodiment, by employing structural reparameterization, the method does not introduce additional computational overhead during training but enhances the model's ability to extract X-ray image features, while simultaneously saving time during foreign object detection during inference.

[0056] To facilitate the use of structural reparameterization and to enable the model to focus on both small, detailed texture features and shape features, making the extracted features more expressive, this embodiment ultimately adopts a structure where the first three stages use convolution and the last stage uses a Transformer Encoder structure.

[0057] It should be noted that, compared to the MAE model, the Encoder in this embodiment adopts a multi-level structure, so traditional masking strategies cannot be used directly. This is because: if the features extracted from stage 1 are directly randomly masked, each token in stage 3 will have some visible information. Therefore, RepConvMAE performs random masking on the input of the last stage (i.e., stage 4), and then upsamples the resulting mask of the last stage (i.e., stage 4) by different factors to obtain the masks of the first three stages. For example, the stage 4 mask is upsampled by 2, 4, and 8 times to obtain the masks of stage 1, stage 2, and stage 3, respectively. The mask of each patch is represented by 0 or 1, where 0 represents masked and 1 represents unmasked.

[0058] Example 3

[0059] Based on the above embodiments, as one possible implementation method, such as Figure 2 As shown, the Decoder in the RepConvMAE self-supervised model includes a Linear layer and a Transformer Block; wherein, the multi-scale feature maps are added after strided convolution and then fed into the Linear layer, and the output of the Linear layer is padded with masked patches and then fed into the Transformer Block.

[0060] For example, the feature maps obtained from each stage of the Encoder have scales of {1 / 4, 1 / 8, 1 / 16, 1 / 32}. The feature maps extracted from the first three stages are added to the feature map of the last stage after strided convolution, and then fed into the Transformer Block after passing through the Linear layer.

[0061] As one possible implementation method, the processing procedure of the Linear layer is represented by formula (2):

[0062] E d =Linear(StrideConv(E1,8)+StrideConv(E2,4)+StrideConv(E3,2)+E4) (2)

[0063] Among them, E dThe output of the Linear layer is represented by StrideConv(,X), which represents a convolution with a stride of X. E1, E2, E3, and E4 represent feature maps of four different scales arranged from largest to smallest. For example, based on the example above, E1, E2, E3, and E4 are the outputs of stage1, stage2, stage3, and stage4, respectively.

[0064] Example 4

[0065] The FPN layer in the EFFR-CNN detection model can use traditional FPN for feature fusion. In the foreign object detection process in textile, footwear, and bag manufacturing workshops, apart from some larger foreign objects that need to be detected, metal foreign objects such as broken needles are often relatively small, generally ranging from 0.4mm to 1.0cm. However, traditional FPN has poor performance in detecting some small objects. Therefore, in order to further improve its detection performance, this embodiment also provides an improved FPN layer.

[0066] For small object detection, each layer of the multi-scale feature map needs to focus not only on the target at that scale, but also help other layers obtain more training samples. To control the priority of these two requirements, a fusion factor is added to the improved FPN layer, specifically: the improved FPN uses formula (3) to aggregate adjacent feature layers;

[0067]

[0068] in, This indicates that a 1x1 convolutional layer achieves channel matching, F up This indicates a doubling upsampling operation to achieve feature map size matching. Conv() represents the convolution operation used for feature processing, α is the fusion factor, and P... i This represents the i-th feature layer;

[0069] Furthermore, it can be understood that the fusion factor is used to describe the coupling degree between adjacent layers of the FPN, and there exists an optimal range of values. In order to obtain an ideal α so that the detection model has the best performance in small target detection, this embodiment of the invention uses the statistical information of the dataset to calculate α, specifically: the value of the fusion factor α is determined by formula (4):

[0070]

[0071] in, and P i+1 Layers and P i The number of objects on the layer.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for foreign object detection in X-ray images based on self-supervised learning, characterized in that, The method comprises the following steps: Step 1: constructing a ConvMAE model based on structural reparameterization, denoted as RepConvMAE self-supervised model, comprising an Encoder for extracting a multi-scale feature map of an input image and a Decoder for reconstructing the image based on the extracted multi-scale feature map; Step 2: pre-training the RepConvMAE self-supervised model using an X-ray image dataset; Step 3: constructing a detection model EFFR-CNN comprising, from shallow to deep, a backbone network, an FPN layer, a shared feature layer, an RPN layer, an ROI Pooling layer and a fully connected layer; the backbone network adopts the Encoder in the pre-trained RepConvMAE self-supervised model; Step 4: training the detection model EFFR-CNN using an X-ray foreign body image dataset; Step 5: inputting a to-be-detected X-ray image into the trained detection model EFFR-CNN to obtain a detection result. 2.The X-ray image foreign object detection method based on self-supervised learning according to claim 1, wherein, The Encoder in the RepConvMAE self-supervised model comprises four stages from shallow to deep; the first three stages have the same structure and each comprises a Patch Embedding layer and a Masked Convolution Block based on structural reparameterization; the last stage comprises a Patch Embedding layer and a Transformer Block; The Masked Convolution Block based on structural reparameterization comprises, from shallow to deep, a BatchNorm layer, a 1*1 convolution layer, a K*K convolution layer based on structural reparameterization, a Batch Norm layer, a 1*1 convolution layer and a 1*1 convolution layer; wherein the K*K convolution layer based on structural reparameterization refers to replacing the K*K convolution layer with three branches during training, namely a 1*K convolution layer followed by a Batch Norm layer, a K*1 convolution layer followed by a Batch Norm layer, and a K*K convolution layer followed by a Batch Norm layer; during inference, the three branches are replaced with a K*K convolution layer. 3.The X-ray image foreign object detection method based on self-supervised learning according to claim 2, characterized in that, The mask strategy of the RepConvMAE self-supervised model is specifically that random masks are directly performed on the input of the last stage, and then the masks of the last stage obtained are upsampled by different multiples to obtain the masks of the first three stages; wherein the mask of each patch is expressed by 0 or 1, 0 represents a mask, and 1 represents an unmasked. 4.The X-ray image foreign object detection method based on self-supervised learning according to claim 1, wherein, The Decoder in the RepConvMAE self-supervised model comprises a Linear layer and a Transformer Block; wherein the multi-scale feature map is added after cross-step convolution and sent to the Linear layer, and the output of the Linear layer is sent to the Transformer Block after the Masked patchs are completed. 5.The self-supervised learning based X-ray image foreign object detection method of claim 4, wherein, The processing process of the Linear layer is represented by formula (2): E d = Linear(StrideConv(E1, 8) + StrideConv(E2, 4) + StrideConv(E3, 2) + E4) (2) where E d is the output of the Linear layer, StrideConv(, X) denotes a convolution with stride X, and E1, E2, E3, E4 denote four feature maps of different sizes arranged in descending order. 6.The self-supervised learning based X-ray image foreign object detection method of claim 1, wherein, The FPN adopts an improved FPN; the improved FPN refers to adding a fusion factor in the original FPN, and specifically includes that the improved FPN adopts formula (3) to aggregate adjacent feature map layers; wherein, represents a convolution layer of 1*1 to realize channel matching, F up represents a two times up-sampling operation to realize feature map size matching, Conv() represents a convolution operation for feature processing, and a is a fusion factor, P i represents an i-th feature map layer. 7.The self-supervised learning based X-ray image foreign object detection method of claim 6, wherein, The value of the fusion factor alpha is determined by formula (4): wherein, and respectively represent P i+1 layer and P i number of objects on the P

Images