Foreign body detection method based on deformable convolution in chest X-ray images
By using a convolutional neural network based on deformable convolution, the problem of foreign body detection in chest X-ray images has been solved, achieving efficient and accurate identification of foreign bodies and improving the accuracy of medical diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE FIRST AFFILIATED HOSPITAL OF ZHEJIANG CHINESE MEDICAL UNIVERSITY
- Filing Date
- 2023-07-03
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies are not effective at detecting foreign bodies in chest X-ray images, especially foreign bodies of different shapes and sizes. Furthermore, the low contrast makes it difficult to distinguish the details of foreign bodies, which can easily lead to missed or false detections.
A convolutional neural network based on deformable convolution, including an improved MoblieNet backbone, spatial pyramid pooling structure, and spatial aggregation module, is used to enhance contrast through image preprocessing, construct multi-scale feature maps, and fuse location information for foreign object prediction.
It improves the accuracy and precision of foreign body detection, reduces missed and false detections, and enhances the accuracy of medical diagnosis.
Smart Images
Figure CN116823786B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a method for foreign body detection in chest X-ray images based on deformable convolution. Background Technology
[0002] Chest X-rays are an important method for detecting heart and lung diseases. Chest images are among the most commonly used medical images and are widely applied in the diagnosis and treatment of cardiothoracic diseases. Early and accurate detection of foreign objects on chest X-ray images can help clinicians improve the accuracy of medical diagnoses.
[0003] Current research on foreign body detection on chest X-ray images is limited, and existing methods cannot effectively detect foreign bodies of different shapes and sizes in real time. To help physicians address this clinical diagnostic challenge, it is essential to research and develop methods for foreign body detection on chest X-ray images.
[0004] With the innovation and gradual maturation of deep learning technology, deep learning-based methods have begun to shine in the field of medical imaging, greatly improving the intelligence of medical diagnosis. However, due to the complexity of medical images, directly applying common deep learning methods to foreign body detection in chest X-ray images mainly presents two problems: First, foreign bodies of different shapes and sizes may appear in the image, increasing the detection difficulty for the detector, especially for smaller foreign bodies. Second, the low contrast of chest X-ray images makes it difficult to display the details of foreign bodies, and low-density foreign bodies are difficult to distinguish from the background, potentially leading to missed or false detections. Summary of the Invention
[0005] This invention addresses the problems of missed and false detections of foreign objects by providing a method for foreign object detection in chest X-ray images based on deformable convolution.
[0006] The technical solution adopted by this invention to solve the technical problem is as follows:
[0007] This invention includes the following steps:
[0008] Step 1: X-ray image preprocessing to enhance image contrast and clarity;
[0009] Step 2: Construct a convolutional neural network based on deformable convolution;
[0010] The convolutional neural network includes an improved MoblieNet backbone network, a spatial pyramid pooling structure, a spatial aggregation module, and a foreign object prediction head.
[0011] Step 3: Input the training set data into a convolutional neural network based on deformable convolution for training to obtain a trained network model;
[0012] Step 4: Input the test set data into the trained network model to obtain the extracted feature map, and then perform prediction and regression to obtain the coordinates of the foreign object.
[0013] Preferably, in step 1, the X-ray image is preprocessed using a contrast-limited adaptive histogram equalization algorithm.
[0014] Preferably, in step 2, the backbone network performs feature extraction to generate a multi-scale feature map; the spatial pyramid structure performs multi-scale feature fusion; and the spatial aggregation module aggregates location information into the features for precise location of the foreign object.
[0015] Preferably, the backbone network is a MobileNet network with deformable convolution, including a first processing unit, a second processing unit, and a third processing unit;
[0016] The first processing unit outputs the first layer feature map to the second processing unit and the spatial aggregation module;
[0017] The second processing unit outputs the second layer feature map to the third processing unit and the spatial aggregation module;
[0018] Preferably, the first processing unit consists of five depth-separable convolutional layers, the second processing unit consists of four depth-separable convolutional layers and deformable convolutional units, and the third processing unit consists of two deformable convolutional layers.
[0019] in:
[0020] Each of the depthwise separable convolutional layers consists of depthwise convolution, pointwise convolution, batch normalization, and ReLU activation function;
[0021] The deformable convolutional unit first predicts the offset of each pixel based on the information in the first layer output feature map. The offset is predicted using a convolutional layer of the same size as the first layer output feature map. Then, the shape of the convolutional kernel is obtained by sampling based on the offset. Finally, the second layer output feature map is output through convolution operation.
[0022] The second layer output feature map is processed by the third processing unit to obtain the feature map output by the backbone network.
[0023] Preferably, the spatial pyramid structure is used to perform four different sizes of max pooling on the feature map output by the backbone network. After the pooling extracts the features, the features of each path are stacked to obtain the third layer of output feature map.
[0024] Preferably, the spatial aggregation module is used to embed spatial location information into image features.
[0025] Preferably, the spatial aggregation module takes a first-layer output feature map, a second-layer output feature map, and a third-layer output feature map as inputs; and outputs three layers of feature maps at different scales.
[0026] For each input feature map layer, it is adjusted to the same feature map size by upsampling or downsampling;
[0027] After concatenating the channels, keep the number of channels unchanged and shuffle them in space to obtain a sequence;
[0028] This sequence is mapped to a two-dimensional feature map by reshaping and edge padding, and then spatially reorganized.
[0029] By performing depthwise convolution operations in space, features are extracted from the two-dimensional feature map, and spatial location information is fused into the image features;
[0030] The convolution operation maps the two-dimensional feature map to the original dimension output.
[0031] Preferably, the foreign object prediction head includes a convolutional branch and a prediction branch;
[0032] The input to the convolutional branch is feature maps of different scales output by the spatial aggregation module; the output is a tensor, which is input to the prediction branch to predict the target.
[0033] The prediction branch contains a convolutional layer and a reshape layer, which are used to convert the feature map into the object detection result.
[0034] Preferably, the convolutional layer is used to convert each pixel in the feature map into a prediction box;
[0035] The reshape layer is used to integrate the coordinates and confidence scores of the predicted bounding boxes.
[0036] The beneficial effects of this invention are as follows: The deformable convolution-based backbone network in this invention can adaptively adjust the offset of features, thereby extracting the features of foreign objects more accurately; the spatial aggregation module in this invention embeds spatial location information into image features, which can better capture the location information of foreign objects. Attached Figure Description
[0037] Figure 1 This is a flowchart of the foreign body detection process in chest X-ray images according to the present invention;
[0038] Figure 2 This is a diagram of the convolutional neural network structure based on deformable convolution of the present invention;
[0039] Figure 3 This is a diagram of the depth-separable convolution structure of the present invention;
[0040] Figure 4 This is a schematic diagram of data preprocessing according to the present invention;
[0041] Figure 5 This is a schematic diagram of the foreign object detection location according to the present invention. Detailed Implementation
[0042] like Figure 1 As shown in the figure, this application provides a method for foreign body detection in chest X-ray images based on deformable convolutional neural networks, including the following steps:
[0043] Step 1: Perform chest X-ray image data preprocessing. The original image is preprocessed using the Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm to enhance image contrast. See [link to relevant documentation]. Figure 4 .
[0044] Step 2: Construct a convolutional neural network based on deformable convolution, which includes four parts: an improved MoblieNet backbone network, a spatial pyramid pooling structure, a spatial aggregation module, and a foreign object prediction head.
[0045] The backbone network extracts features to generate multi-scale feature maps, the spatial pyramid structure performs multi-scale feature fusion, and the spatial aggregation module aggregates location information into features to solve the problem of inaccurate foreign object location.
[0046] like Figure 2 As shown, in one embodiment, the first part is an improved MobileNet backbone network, i.e., a MobileNet network with deformable convolutions. The preprocessed image, after normalization, is input into the network and passes through a standard convolutional layer, the purpose of which is to extract low-level features from the image. This layer uses a 3x3 convolutional kernel with a stride of 2 and 32 output channels. This convolutional layer performs convolution operations on the input image to obtain a feature map. This feature map C1 is then obtained by sequentially passing through five depthwise separable convolutional layers.
[0047] In a preferred embodiment, such as Figure 3 As shown, each depthwise separable convolutional layer consists of four parts: depthwise convolution, pointwise convolution, batch normalization, and ReLU activation. The depthwise convolutional layer uses a 3x3 kernel with a stride of 1. Unlike ordinary convolutional layers, the depthwise convolutional layer only convolves one channel, rather than all channels. This reduces computation and overfitting. The pointwise convolutional layer uses a 1x1 kernel to convolve each channel, fusing information from different channels. Batch normalization and ReLU activation, performed five times, gradually extract higher-level features.
[0048] like Figure 2As shown, feature map C1 is then passed through four depthwise separable convolutional layers followed by two deformable convolutional layers (these two constitute the deformable convolutional unit). The deformable convolutional unit first predicts the offset of each pixel based on the information in feature map C1. The offset prediction uses a convolutional layer of the same size as feature map C1. Then, the shape of the convolutional kernel is obtained by sampling based on the offset, and finally, the output feature map C2 is obtained through a convolution operation. The deformable convolutional layer can better adapt to the shape of the target, thereby improving the accuracy of the model.
[0049] Feature map C2 is then passed through two deformable convolutional layers to obtain the first part of the final feature map. Feature maps C1, C2, and the first part of the final feature map differ in scale and number of channels, thus obtaining feature maps of different scales for better multi-scale prediction in subsequent steps.
[0050] In one embodiment, the second part of the spatial pyramid structure is obtained by performing max pooling on the feature map of the final output of the first part with four different sizes: 1×1, 5×5, 9×9, and 13×13. After extracting features by pooling, the features of each path are stacked to obtain the feature map C3.
[0051] In one embodiment, the third spatial aggregation module takes feature maps C1, C2, and C3 as input. For each feature map layer, it is first adjusted to the same feature map size through upsampling or downsampling, and then channel concatenation is performed while maintaining the same number of channels. The resulting sequence is shuffled in space, and then reshaped and padded to map this sequence into a two-dimensional feature map. This spatial recombination process ensures that the spatial location information of the feature map is represented. Padded ensures that the size of the feature map remains unchanged during spatial recombination, maintaining consistency and preventing the loss of channel features. Next, depthwise convolution is performed in space to extract features from the two-dimensional feature map, fusing the spatial location information into the image features. Finally, ordinary convolution is used to map the two-dimensional feature map back to the original dimension for output. Thus, this module outputs three feature maps M1, M2, and M3 at different scales.
[0052] Specifically, in combination Figure 2For feature maps C1 and C2, 1x1 convolutions are first applied to halve the number of channels. For feature map C3, an upsampling operation is performed to resize it to the same size as feature map C2. Then, channel concatenation is performed with feature map C2 to obtain a new feature map. This new feature map is then shuffled in space to obtain a sequence. Reshaping and padding are used to map this sequence to a two-dimensional feature map, resulting in spatial recombination. Next, depthwise convolution is performed in space to extract features from the two-dimensional feature map. Finally, a regular convolution operation is used to map the two-dimensional feature map back to its original dimension, resulting in feature map P2. Feature map P2 is then upsampled to the same size as feature map C1, and channel concatenation is performed with feature map C1 to obtain a new feature map. This new feature map is then shuffled in space to obtain a sequence. Reshaping and padding are used to map this sequence to a two-dimensional feature map, resulting in spatial recombination. Next, feature extraction is performed on the two-dimensional feature map by performing a depthwise convolution operation in space. Finally, a regular convolution operation is used to map the two-dimensional feature map back to the original dimension, resulting in feature map P1. Feature map P1 is then subjected to a 3x3 convolution with a stride of 2 to reduce its size by half. It is then concatenated with feature map P2 through channels to obtain feature map M2. Feature map M2 is then subjected to a 3x3 convolution with a stride of 2 to reduce its size by half. It is then concatenated with feature map C3 through channels to obtain feature map M3. Feature map M1 is feature map P1.
[0053] Step 3: Input the training set data into a convolutional neural network based on deformable convolution for training to obtain a trained network model;
[0054] Step 4: Input the test set data into the trained network model to obtain the extracted feature map, and then perform prediction and regression to obtain the coordinates of the foreign object.
[0055] In one embodiment, each of the foreign object prediction heads includes:
[0056] Convolutional branch: The feature maps of different scales output by the spatial aggregation module are processed by a convolutional block, which includes 3x3 convolution, group normalization, and Leaky ReLU function to obtain a tensor. This tensor is then fed into a prediction branch to predict the target.
[0057] The prediction branch contains a convolutional layer and a reshape layer, used to transform the feature map into object detection results. Specifically, the convolutional layer converts each pixel in the feature map into a predicted bounding box, while the reshape layer integrates the coordinates and confidence scores of the predicted bounding boxes.
[0058] Finally, the three foreign object prediction heads, through three prediction branches of different scales, can obtain target detection results at three different scales.
[0059] This application also provides verification examples:
[0060] A dataset provided by a provincial people's hospital was selected to train a convolutional neural network model and evaluate its performance. The dataset contains 3375 images with foreign objects. According to the training set:validation set ratio of 9:1, 3071 images were used for training, 304 images with foreign objects and 300 images without foreign objects were used for validation, and the remaining 662 images were used for the test set.
[0061] To reduce the number of parameters, all images in the dataset were re-cropped to 608×608, and mosaic data augmentation was used, along with the SGD optimizer. The threshold was set to 0.5, the weight decay rate to 0.0005, the maximum learning rate to 0.01, and the minimum learning rate to 0.01. Validation was performed using validation set data after each training epoch, and finally, the detection performance of the model was evaluated using a separate test set after training.
[0062] The training set data is input into the aforementioned convolutional neural network based on deformable convolution to obtain a trained network model; the test set data is input into the trained network model to obtain extracted feature maps, which are then used for prediction and regression to obtain the coordinates of the foreign object, such as... Figure 5 As shown.
[0063] On this test set, the mean accuracy (mAP) reached 0.7188, which is higher than the mAP of 0.6802 of the YOLOv4 method, representing an improvement of about 4%.
Claims
1. A method for foreign body detection in chest X-ray images based on deformable convolution, characterized in that... The method includes the following steps: Step 1: X-ray image preprocessing to enhance image contrast and clarity; Step 2: Construct a convolutional neural network based on deformable convolution; The convolutional neural network includes an improved MobileNet backbone, a spatial pyramid pooling structure, a spatial aggregation module, and a foreign object prediction head. Step 3: Input the training set data into a convolutional neural network based on deformable convolution for training to obtain a trained network model; Step 4: Input the test set data into the trained network model to obtain the extracted feature map, and then perform prediction and regression to obtain the coordinates of the foreign object; The backbone network is a MobileNet network with deformable convolutions, including a first processing unit, a second processing unit, and a third processing unit; The first processing unit outputs the first layer feature map to the second processing unit and the spatial aggregation module; The second processing unit outputs the second layer feature map to the third processing unit and the spatial aggregation module; The first processing unit consists of five depthwise separable convolutional layers; the second processing unit consists of four depthwise separable convolutional layers and deformable convolutional units; the third processing unit consists of two deformable convolutional layers. in: Each of the depthwise separable convolutional layers consists of depthwise convolution, pointwise convolution, batch normalization, and ReLU activation function; The deformable convolutional unit first predicts the offset of each pixel based on the information in the first layer output feature map. The offset is predicted using a convolutional layer of the same size as the first layer output feature map. Then, the shape of the convolutional kernel is obtained by sampling based on the offset. Finally, the second layer output feature map is output through convolution operation. The second layer output feature map is processed by the third processing unit to obtain the feature map output by the backbone network; The spatial aggregation module is used to embed spatial location information into image features; The spatial aggregation module takes three output feature maps as inputs: a first-layer output feature map, a second-layer output feature map, and a third-layer output feature map; and outputs three layers of feature maps at different scales. For each input feature map layer, it is adjusted to the same feature map size by upsampling or downsampling; After concatenating the channels, keep the number of channels unchanged and shuffle them in space to obtain a sequence; This sequence is mapped to a two-dimensional feature map by reshaping and edge padding, and then spatially reorganized. By performing depthwise convolution operations in space, features are extracted from the two-dimensional feature map, and spatial location information is fused into the image features; The convolution operation maps the two-dimensional feature map to the original dimension output.
2. The method according to claim 1, characterized in that: In step 1, the X-ray image is preprocessed using a contrast-limited adaptive histogram equalization algorithm.
3. The method according to claim 1, characterized in that: In step 2, the backbone network performs feature extraction to generate multi-scale feature maps; the spatial pyramid structure performs multi-scale feature fusion; and the spatial aggregation module aggregates location information into features for precise location of foreign objects.
4. The method according to claim 1, characterized in that: The spatial pyramid structure is used to perform max pooling of four different sizes on the feature map output by the backbone network. After the pooling extracts the features, the features of each path are stacked to obtain the third layer of output feature map.
5. The method according to claim 1, characterized in that: The foreign object prediction head includes a convolutional branch and a prediction branch; The input to the convolutional branch is feature maps of different scales output by the spatial aggregation module; the output is a tensor, which is input to the prediction branch to predict the target. The prediction branch contains a convolutional layer and a reshape layer, which are used to convert the feature map into the object detection result.
6. The method according to claim 5, characterized in that: The convolutional layer is used to convert each pixel in the feature map into a prediction box; The reshape layer is used to integrate the coordinates and confidence scores of the predicted bounding boxes.