X-ray image lesion detection method and device

Abstract

This invention discloses a method and apparatus for detecting lesions in X-ray images. The method includes: an X-ray image feature extraction module for extracting image features from X-ray images; an edge-guided spinal feature enhancement module for extracting spinal edge maps from X-ray images using an edge detector and enhancing spinal features using the spinal edge maps; a candidate feature extraction module for obtaining candidate features from the enhanced spinal features; a context-aggregated structural relationship reasoning module for obtaining query features and context features from the enhanced spinal features, mining the correlation between the query features and context features to obtain aggregated features, and using the aggregated features to compensate for the candidate features; and a spinal lesion detection head for detecting spinal lesions using the compensated candidate features. The apparatus includes: a processor and a memory. This invention delves into the characteristics of spinal lesions in X-ray images to achieve more accurate lesion perception, thereby improving the performance of spinal lesion detection.

Description

Technical Field

[0001] This invention relates to the field of X-ray image detection, and more particularly to a method and apparatus for detecting lesions in X-ray images. Background Technology

[0002] The spine, as a vital component of the human skeletal system, plays a crucial role in protecting the central nervous system and maintaining balance. Because spinal diseases can cause serious consequences such as nerve damage and limb dysfunction, accurate diagnosis is essential. Doctors need to conduct in-depth analysis of spinal images to understand the extent of lesions during diagnosis. Spinal lesion detection technology aims to automatically identify the type and location of spinal lesions from images, assisting doctors in diagnosing spinal diseases. Therefore, research on spinal lesion detection technology is of great significance.

[0003] Due to its advantages such as low radiation and ease of operation, X-ray imaging has become one of the important bases for diagnosing spinal diseases. In recent years, thanks to the powerful feature extraction capabilities of deep learning, deep learning-based X-ray image spinal lesion detection technology has attracted widespread attention from researchers. Nguyen et al. proposed a two-stage spinal lesion detection method. This method first uses a densely connected network, DenseNet, to determine whether spinal lesions are present in X-ray images. Then, it further detects images with lesions to determine the type and location of the lesions, thereby effectively improving the accuracy of spinal lesion detection. A method for identifying lumbar spondylolisthesis was proposed by researchers. This method first locates the lumbar region of the spine in an X-ray image, and then uses an image classification network to determine whether lumbar spondylolisthesis has occurred in the lumbar region.

[0004] The methods described above directly utilize detection techniques found in natural images to detect spinal lesions in X-ray images. However, these methods do not fully consider the characteristics of spinal lesions in X-ray images, resulting in limited performance in spinal lesion detection. Therefore, further exploration is needed to explore how to utilize the characteristics of spinal lesions in X-ray images to improve the accuracy of lesion detection. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting lesions in X-ray images. This invention fully utilizes the characteristics of spinal lesions in X-ray images to improve lesion detection performance, enhances spinal features using spinal edge information to obtain more discriminative feature representations, and simultaneously explores the structural relationships between spinal vertebrae to achieve more accurate lesion perception, thereby improving the performance of spinal lesion detection. See the description below for details:

[0006] A first aspect: a method for detecting lesions using X-ray imaging, the method comprising:

[0007] The X-ray image feature extraction module takes an X-ray image as input and obtains image features.

[0008] The edge-guided spine feature enhancement module takes X-ray images and image features as input, extracts spine edge maps from X-ray images using histogram equalization, median filtering, and a Canny edge detector; extracts edge features from the spine edge maps using an edge map encoder; obtains edge attention maps corresponding to the image features using activation layers and multi-scale downsampling layers; and enhances the spine features using the edge attention maps to obtain enhanced spine features.

[0009] The candidate feature extraction module takes the enhanced spine features and N randomly initialized learnable candidate boxes as input, and uses RoIAlign to obtain candidate features from the enhanced spine features.

[0010] The context aggregation structural relationship reasoning module takes the highest-level features of the enhanced spine features, N randomly initialized learnable candidate boxes and candidate features as input. It obtains the context features corresponding to the candidate features in the highest-level features through the context feature extraction unit; obtains the aggregated features corresponding to the candidate features through the feature aggregation unit; and adds the aggregated features to the candidate features through the feature compensation unit to obtain the compensated candidate features.

[0011] The spinal lesion detection head uses compensated candidate features and N randomly initialized learnable features as input to complete the detection of spinal lesions.

[0012] The edge-guided spinal feature enhancement module is represented as follows:

[0013]

[0014] in, This indicates histogram equalization, and MF(·) indicates median filtering. This represents the Canny edge detector; the obtained spine edge map I e The data is fed into an edge map encoder to extract edge features f. e , with edge features f e As input, activation layers and multi-scale downsampling layers are used to obtain features from X-ray images. Edge attention maps of the same size

[0015]

[0016] Among them, E D(·) represents the edge map encoder, S(·) represents the activation layer, and MSDL(·) represents the multi-scale downsampling layer. The edge attention map is used to weight the X-ray image features to obtain enhanced spine features.

[0017]

[0018] Here, ⊙ represents element-wise multiplication.

[0019] The context feature extraction unit is defined as follows: based on the learnable candidate boxes in the high-level feature P... M Obtain query features

[0020]

[0021] Where RA(·) represents RoIAlign;

[0022] In the highest level feature P M Evenly spread reference positions in the spatial dimension Reference position With query features Correspondingly, the reference position is obtained through offset learning operations. The corresponding offset is used to calculate the offset position based on the reference position and the offset.

[0023]

[0024] Where, θ offset (·) indicates the offset learning operation;

[0025] Based on the obtained offset position Through bilinear interpolation sampling operation at P M Sampling is performed to obtain context features.

[0026]

[0027] Where φ(·) represents the bilinear interpolation sampling operation.

[0028] The feature aggregation unit is:

[0029] Using linear mapping W q query features Mapping to query vector Using linear mapping W K and W V Context features respectively Mapping to key vector Sum value vector

[0030]

[0031] Based on the obtained key vector Sum value vector Calculate the correlation matrix A;

[0032]

[0033] Where σ(·) represents the softmax layer, and d represents Channel dimension;

[0034] Aggregation features are obtained under the guidance of correlation matrix A.

[0035]

[0036] in, This indicates a pixel addition operation.

[0037] The feature compensation unit is configured to add aggregated features to candidate features to compensate for the candidate features, thereby obtaining compensated candidate features.

[0038]

[0039] in, This indicates a pixel addition operation.

[0040] In a second aspect, an X-ray imaging lesion detection device includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to execute the method described in the first aspect.

[0041] The beneficial effects of the technical solution provided by this invention are:

[0042] 1. This invention utilizes the characteristics of spinal lesions in X-ray images to effectively improve the performance of spinal lesion detection;

[0043] 2. This invention designs an edge-guided spinal feature enhancement module, which utilizes spinal edge information to enhance spinal features, thereby obtaining a more discriminative feature representation;

[0044] 3. This invention designs a context-aggregated structural relationship reasoning module, which delves into the structural relationships between vertebrae of the spine, thereby achieving more accurate lesion perception. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of an X-ray imaging lesion detection method. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0047] Example 1

[0048] The following examples illustrate the specific implementation of an X-ray image lesion detection method according to an embodiment of the present invention.

[0049] I. Constructing an X-ray Image Feature Extraction Module

[0050] The X-ray image feature extraction module takes an X-ray image as input and aims to extract multi-scale features from the image. Specifically, the module extracts M layers of X-ray image features. Where F m Let M represent the learned X-ray image features of the m-th layer, with M set to 4. The feature extraction formula is as follows:

[0051]

[0052] Among them, E F (·) indicates the X-ray image feature extraction module. This module consists of ResNet and FPN networks. ResNet and FPN networks are well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0053] II. Design of an edge-guided spinal feature enhancement module

[0054] Based on the X-ray image feature extraction module, this embodiment of the invention designs an edge-guided spine feature enhancement module. This module uses X-ray images and image features... The input is used to obtain enhanced spinal features. Specifically, the designed edge-guided spinal feature enhancement module first extracts the spinal edge map I from the X-ray image. e Specifically, histogram equalization is first used to enhance the spine edges, then median filtering is used to reduce noise in the X-ray image, and finally the Canny edge detector is used to extract the spine edges. The formula for the above process is as follows:

[0055]

[0056] in, This indicates histogram equalization, and MF(·) indicates median filtering. This refers to the Canny edge detector.

[0057] Then, the obtained spinal edge map I e The data is fed into an edge map encoder to extract edge features f. eThen, using edge features f e As input, activation layers and multi-scale downsampling layers are used to obtain features from X-ray images. Edge attention maps of the same size The formula is expressed as follows:

[0058]

[0059] Among them, E D (·) represents the edge map encoder, which consists of three convolutional layers with a kernel size of 1×1. S(·) represents the activation layer, which is implemented using the sigmoid activation function. MSDL(·) represents the multi-scale downsampling layer, which consists of four different scales of max pooling operations. The max pooling kernels are 4, 8, 16, and 32, respectively.

[0060] Finally, using the obtained edge attention map X-ray image features Weighting is performed to obtain enhanced spinal features. The formula is as follows:

[0061]

[0062] Here, ⊙ represents element-wise multiplication.

[0063] III. Constructing a Candidate Feature Extraction Module

[0064] To obtain the potential location of the lesion and the features of the corresponding region, this embodiment of the invention constructs a candidate feature extraction module. This module uses enhanced spinal features... and N randomly initialized learnable candidate boxes As input, RoIAlign is used to extract enhanced spinal features. Obtain candidate features N is set to 100. The formula is as follows:

[0065]

[0066] Wherein, RA(·) represents RoIAlign, and the feature space resolution extracted by RoIAlign is set to 7×7. RoIAlign is well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0067] IV. Design a structural relationship reasoning module for context aggregation

[0068] To fully explore the structural relationships between vertebral bodies in the spine and improve the accuracy of spinal lesion detection, this invention incorporates a context-aggregated structural relationship reasoning module. This module utilizes the highest-level feature P of the enhanced spinal features. MN randomly initialized learnable candidate boxes and candidate features As input, the highest-level feature P is first obtained through the context feature extraction unit. M Middle and candidate features Corresponding context features Then, the feature aggregation unit is used to obtain candidate features. Corresponding aggregate features Finally, the feature compensation unit is used to aggregate the features. Add candidate features In the process, the candidate features after compensation are obtained.

[0069] In the context feature extraction unit, the first step is to extract learnable candidate boxes. In high-level feature P M Obtain query features

[0070]

[0071] Where RA(·) represents RoIAlign.

[0072] Then, in the highest level feature P M Evenly spread reference positions in the spatial dimension Reference position With query features Correspondingly, the reference position is then obtained through offset learning operations. The corresponding offset is used to calculate the offset position based on the reference position and the offset.

[0073]

[0074] Where, θ offset (·) indicates the offset learning operation. This operation consists of two 3×3 convolutional layers, a GeLU activation layer, and a normalization layer.

[0075] Finally, based on the obtained offset position Through bilinear interpolation sampling operation at P M Sampling is performed to obtain context features. The formula is as follows:

[0076]

[0077] Where φ(·) represents the bilinear interpolation sampling operation.

[0078] In the feature aggregation unit, the linear mapping W is first used. q query features Mapping to query vector And using linear mapping W K and W V Context features respectively Mapping to key vector Sum value vector

[0079]

[0080] Then, based on the obtained key vector Sum value vector Calculate the correlation matrix A.

[0081]

[0082] Where σ(·) represents the softmax layer, and d represents The channel dimension is set to 256.

[0083] Finally, aggregated features are obtained under the guidance of the correlation matrix A. The formula is expressed as follows:

[0084]

[0085] in, This indicates a pixel addition operation.

[0086] In the feature compensation unit, aggregated features will be... Add candidate features China and Israel achieve candidate feature The compensation is used to obtain the compensated candidate features. The formula is as follows:

[0087]

[0088] in, This indicates a pixel addition operation.

[0089] V. Constructing a Spinal Lesion Detection Head

[0090] Spinal lesion detection head with compensated candidate features and N randomly initialized learnable features As input, spinal lesion detection is performed. Specifically, classification and regression branches are used to predict the lesion category, respectively. and the location of the lesion The formula is expressed as follows:

[0091]

[0092] in, Indicates a category branch, This indicates the regression branch.

[0093] Finally, the predicted lesion categories and the location of the lesion The results are combined to obtain the lesion detection results, and the final lesion detection result is obtained through non-maximum suppression operation.

[0094] VI. Training a spinal lesion detection network

[0095] This invention proposes an X-ray image lesion detection network to optimize the spinal lesion detection process in an end-to-end manner. First, the Hungarian matching algorithm is used to match the predicted results with the actual lesions. Then, an ensemble prediction loss is used to constrain the distance between the predicted results and the actual lesions. The formula for the ensemble prediction loss is as follows:

[0096] L=λ cls ·L cls +λ L1 ·L L1 +λ giou ·L giou

[0097] Among them, L cls L represents Focal loss. L1 Indicates Smooth L1 loss, L giou λ represents the GIOU loss. cls , λ L1 and λ giou λ represents the coefficients of Focal loss, Smooth L1 loss, and GIOU loss, respectively. In this embodiment of the invention, λ cls , λ L1 and λ giou Set them to 2.0, 5.0 and 2.0 respectively.

[0098] The aforementioned Focal loss, Smooth L1 loss, and GIOU loss are well known to those skilled in the art, and will not be described in detail in the embodiments of the present invention.

[0099] Example 2

[0100] An X-ray imaging lesion detection device includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the following method steps in Embodiment 1:

[0101] Using X-ray images and image features as input, histogram equalization, median filtering, and Cann are employed. yAn edge detector extracts a spine edge map from an X-ray image; an edge map encoder extracts edge features from the spine edge map; an activation layer and a multi-scale downsampling layer are used to obtain an edge attention map corresponding to the image features; and the edge attention map is used to enhance the spine features to obtain enhanced spine features.

[0102] Using enhanced spine features and N randomly initialized learnable candidate boxes as input, RoIAlign is used to obtain candidate features from the enhanced spine features.

[0103] The algorithm takes the highest-level feature of the enhanced spine feature, N randomly initialized learnable candidate boxes and candidate features as input, and obtains the context features corresponding to the candidate features in the highest-level feature through the context feature extraction unit; obtains the aggregated features corresponding to the candidate features through the feature aggregation unit; and adds the aggregated features to the candidate features through the feature compensation unit to obtain the compensated candidate features.

[0104] The spinal lesion detection is completed by using the compensated candidate features and N randomly initialized learnable features as input.

[0105] The edge-guided enhancement of spinal features is represented as follows:

[0106]

[0107] in, This indicates histogram equalization, and MF(·) indicates median filtering. This represents the Canny edge detector; the obtained spine edge map I e The data is fed into an edge map encoder to extract edge features f. e , with edge features f e As input, activation layers and multi-scale downsampling layers are used to obtain features from X-ray images. Edge attention maps of the same size

[0108]

[0109] Among them, E D (·) represents the edge map encoder, S(·) represents the activation layer, and MSDL(·) represents the multi-scale downsampling layer. The edge attention map is used to weight the X-ray image features to obtain enhanced spine features.

[0110]

[0111] Here, ⊙ represents element-wise multiplication.

[0112] The context feature extraction unit is: based on the learnable candidate boxes in the high-level feature P MObtain query features

[0113]

[0114] Where RA(·) represents RoIAlign;

[0115] In the highest level feature P M Evenly spread reference positions in the spatial dimension Reference position With query features Correspondingly, the reference position is obtained through offset learning operations. The corresponding offset is used to calculate the offset position based on the reference position and the offset.

[0116]

[0117] Where, θ offset (·) indicates the offset learning operation;

[0118] Based on the obtained offset position Through bilinear interpolation sampling operation at P M Sampling is performed to obtain context features.

[0119]

[0120] Where φ(·) represents the bilinear interpolation sampling operation.

[0121] The feature aggregation unit is:

[0122] Using linear mapping W q query features Mapping to query vector Using linear mapping W K and W V Context features respectively Mapping to key vector Sum value vector

[0123]

[0124] Based on the obtained key vector Sum value vector Calculate the correlation matrix A;

[0125]

[0126] Where σ(·) represents the softmax layer, and d represents Channel dimension;

[0127] Aggregation features are obtained under the guidance of correlation matrix A.

[0128]

[0129] in, This indicates a pixel addition operation.

[0130] The feature compensation unit involves adding aggregated features to candidate features to compensate for the candidate features, resulting in compensated candidate features.

[0131]

[0132] in, This indicates a pixel addition operation.

[0133] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0134] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting lesions using X-ray imaging, characterized in that, The method includes: The X-ray image feature extraction module takes an X-ray image as input and obtains image features. The edge-guided spine feature enhancement module takes X-ray images and image features as input, extracts spine edge maps from X-ray images using histogram equalization, median filtering, and a Canny edge detector; extracts edge features from the spine edge maps using an edge map encoder; obtains edge attention maps corresponding to the image features using activation layers and multi-scale downsampling layers; and enhances the spine features using the edge attention maps to obtain enhanced spine features. The candidate feature extraction module takes the enhanced spine features and N randomly initialized learnable candidate boxes as input, and uses RoIAlign to obtain candidate features from the enhanced spine features. The context aggregation structural relationship reasoning module takes the highest-level features of the enhanced spine features, N randomly initialized learnable candidate boxes and candidate features as input. It obtains the context features corresponding to the candidate features in the highest-level features through the context feature extraction unit; obtains the aggregated features corresponding to the candidate features through the feature aggregation unit; and adds the aggregated features to the candidate features through the feature compensation unit to obtain the compensated candidate features. The spinal lesion detection head uses compensated candidate features and N randomly initialized learnable features as input to complete the detection of spinal lesions.

2. The X-ray image lesion detection method according to claim 1, characterized in that, The edge-guided spinal feature enhancement module is represented as follows: in, This indicates histogram equalization, and MF(·) indicates median filtering. This represents the Canny edge detector; the obtained spine edge map I e The data is fed into an edge map encoder to extract edge features f. e , with edge features f e As input, activation layers and multi-scale downsampling layers are used to obtain features from X-ray images. Edge attention maps of the same size Among them, E D (·) represents the edge map encoder, S(·) represents the activation layer, and MSDL(·) represents the multi-scale downsampling layer. The edge attention map is used to weight the X-ray image features to obtain enhanced spine features. Here, ⊙ represents element-wise multiplication.

3. The X-ray image lesion detection method according to claim 1, characterized in that, The context feature extraction unit is: Based on learnable candidate boxes In high-level feature P M Obtain query features Where RA(·) represents RoIAlign; In the highest level feature P M Evenly spread reference positions in the spatial dimension Reference position With query features Correspondingly, the reference position is obtained through offset learning operations. The corresponding offset is used to calculate the offset position based on the reference position and the offset. Where, θ offset (·) indicates the offset learning operation; Based on the obtained offset position Through bilinear interpolation sampling operation at P M Sampling is performed to obtain context features. Where φ(·) represents the bilinear interpolation sampling operation.

4. The X-ray image lesion detection method according to claim 1, characterized in that, The feature aggregation unit is: Using linear mapping W q query features Mapping to query vector Using linear mapping W K and W V Context features respectively Mapping to key vector Sum value vector Based on the obtained key vector Sum value vector Calculate the correlation matrix A; Where σ(·) represents the softmax layer, and d represents Channel dimension; Aggregation features are obtained under the guidance of correlation matrix A. in, This indicates a pixel addition operation.

5. The X-ray image lesion detection method according to claim 4, characterized in that, The feature compensation unit is: Add aggregated features to candidate features The middle part is used to compensate for the candidate features, and the compensated candidate features are obtained: in, This indicates a pixel addition operation.

6. An X-ray imaging lesion detection device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-5.

Images