Hip joint osteophyte detection method based on texture perception and multi-scale feature adaptive fusion

Abstract

The invention relates to the technical field of medical image analysis, computer vision and deep learning, in particular to a texture perception and multi-scale feature adaptive fusion hip joint osteophyte detection method. According to the method, firstly, a bone texture feature extraction module is used for carrying out feature extraction and enhancement on a hip joint X-ray image, a parallel double-branch structure is adopted, gradient and space structure features are extracted through a Sobel operator and pooling operation, and feature maps are fused; then, the fused features are input into a double-backbone network, the first backbone network extracts local fine textures and long-range dependence features by combining convolution, self-attention and a gating mechanism, and the second backbone network extracts multi-scale semantic features through hierarchical grouping and depth separable convolution; and double-trunk output is dynamically weighted and fused through an adaptive fusion module, features are optimized through a multi-scale semantic fusion module, and finally the position and confidence of osteophyte are output.

Description

Technical Field

[0001] This invention relates to the fields of medical image analysis, computer vision, and deep learning, and in particular to a method for detecting hip osteophytes by adaptive fusion of texture perception and multi-scale features. Background Technology

[0002] Osteoarthritis of the hip (OA) is a common chronic disease among the elderly, imposing a significant burden on patients and society. Osteophytes are a typical radiographic feature of OA, and their accurate identification is crucial for early diagnosis and treatment decisions. While X-ray imaging is a commonly used method for diagnosing OA, it suffers from difficulties in image interpretation and a high degree of subjectivity. Therefore, an objective and precise automated technology is needed to improve the accuracy of osteophyte identification.

[0003] Artificial intelligence technology has shown potential in the field of medical image analysis, but its application in hip osteophyte identification faces technical challenges, such as the lack of large-scale labeled datasets, the easy loss of osteophyte features, and the difficulty in distinguishing pathological changes. Existing deep learning methods have shortcomings in osteophyte detection, cannot provide clinical interpretability, and research in the field of hip joints is still insufficient.

[0004] Currently, neither manual image interpretation nor rudimentary deep learning models have effectively solved the problem of accurate detection of osteophytes in hip X-ray images. Therefore, innovative deep learning methods are urgently needed to overcome this challenge and achieve high-precision end-to-end detection. Summary of the Invention

[0005] This invention aims to address the problems of existing methods for detecting osteophytes in hip joint X-ray images, which rely on manual methods, are highly subjective, and have a high rate of missed detection of early, small lesions. It proposes a method for detecting hip joint osteophytes that integrates texture perception and multi-scale feature adaptive fusion, thereby achieving automated and accurate detection and localization of osteophytes and improving the objectivity and consistency of diagnosis.

[0006] Therefore, this invention proposes a method for detecting hip osteophytes by adaptive fusion of texture perception and multi-scale features, comprising the following steps:

[0007] Obtain X-ray images of the hip joint;

[0008] The bone texture feature extraction module extracts and enhances the features of the hip joint X-ray image. The bone texture feature extraction module adopts a parallel dual-branch structure. The first branch uses a fixed-weight edge detection operator to extract gradient features, and the second branch uses pooling operation to extract spatial structure features. The feature maps output by the two branches are then stitched together in the channel dimension to obtain a feature map that integrates edge details and spatial structure.

[0009] The fused feature map is input into a dual backbone network. The first backbone network extracts local fine texture and long-range dependency features through a hybrid structure combining convolution and self-attention and integrating a gating mechanism. The second backbone network extracts and associates multi-scale semantic features through a hierarchical grouping structure and depthwise separable convolution. Here, long-range dependency features refer to the semantic associations and contextual relationships between pixels or regions that are far apart in the image captured by the model.

[0010] The adaptive fusion module dynamically weights and fuses the features output by the dual backbone network. The adaptive fusion module first aligns the channels of the dual backbone output features, then generates pixel-level adaptive weights through a content-aware mechanism, and uses a learnable global scaling factor to calibrate and fuse the weighted features to obtain the fused features.

[0011] The fused features are input into a multi-scale semantic fusion module, which includes an osteophyte feature pyramid network with bidirectional feature pathways. Feature integration and optimization are achieved by channel alignment, attention recalibration and cross-level information interaction of features at different scales.

[0012] Based on the optimized features, osteophyte detection is performed, and the location and confidence level of the osteophytes are output.

[0013] Technical effects of the present invention:

[0014] This invention significantly improves the accuracy and robustness of osteophyte detection through dual-branch texture enhancement, dual-main-branch feature extraction, adaptive weighted fusion, and multi-scale semantic optimization. In particular, it enhances the sensitivity to small, low-contrast lesions, effectively reducing missed and false detections, and providing a reliable and efficient auxiliary diagnostic tool for clinical use. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall architecture of the BTP-StitchNet model in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of the architecture of the bone texture feature network (BFN) module in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of the design architecture of the Stitch fusion module in an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the architecture of the osteophyte feature pyramid network (OFPN) in an embodiment of the present invention;

[0019] Figure 5Detailed schematic diagrams of the hip joint osteophyte (hereinafter also referred to as HipOsteo-Meta) module and the spatial-to-depth convolution (hereinafter also referred to as SPDConv) module according to embodiments of the present invention are shown; wherein, Figure 5 (A) in the diagram details the structure of the convolution gated linear unit (CGLU), which is a structure used to process specific data. Figure 5 (B) shows the HipOsteo-Meta module, an advanced metadata processing module that enables complex analysis and management of data; Figure 5 (C) in the text specifically explains the SPDConv operation, which is a spatial-to-depth convolution;

[0020] Figure 6 This is a heatmap showing the region of interest (ROI) of the BTP-StitchNet model on an X-ray image in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, they will be described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown in the figure, this invention provides a method for detecting hip osteophytes by adaptively fusing texture awareness and multi-backbone networks. It proposes a dual-backbone deep learning architecture for object detection called BTP-StitchNet, the overall architecture of which is as follows: Figure 1 As shown, it includes the following steps:

[0023] Step S1: Low-level feature extraction and enhancement based on bone texture feature network (BFN).

[0024] High-quality low-level feature input is crucial before deep learning models can perform advanced semantic analysis. Osteophyte lesions often manifest as subtle, irregular protrusions on the bone surface, with extremely subtle differences in texture edges compared to normal bone. To effectively capture these key diagnostic clues, this invention designs a Bone Texture Feature Network (BFN), such as... Figure 2 As shown. Figure 2 This is a schematic diagram of the Bone Texture Feature Network (BFN) module in an embodiment of the present invention. This module acts as an information collector, designed to extract edge details and spatial context information complementaryly through parallel processing, providing clear and robust initial feature maps for subsequent networks. The module employs a parallel dual-branch architecture:

[0025] Sobel Branch: This branch aims to enhance the gradient features of osteophyte edges. Osteophytes appear in X-ray images as rapid changes in pixel intensity, i.e., abrupt discontinuities in bone density. The classic Sobel operator is an ideal tool designed to detect such intensity abrupt changes. This embodiment uses a standard 3×3 Sobel kernel, at the level (G x ) and vertical (G) y The input image I is convolved in the direction shown below to calculate the gradient, which is as follows:

[0026] (1)

[0027] (2)

[0028] (3)

[0029] (4)

[0030] In the above formula, Represents the horizontal gradient feature map, quantifying the rate of change of pixel intensity in the horizontal direction of the image. The more obvious the horizontal edge of the osteophyte, the more pronounced its position. The larger the value; This is a vertical gradient feature map, reflecting changes in pixel intensity along the vertical direction. The more pronounced the vertical edge of the osteophyte, the higher the corresponding position. The larger the value. This involves performing two-dimensional convolution operations to extract features from the input image using a sliding window with the Sobel kernel. The input is the raw grayscale X-ray image of the hip joint, with pixel values ​​ranging from 0 to 255, recording the grayscale distribution of bones and soft tissues; To synthesize the gradient amplitude feature map, the overall significance of the bone edge is quantified, and the greater the difference in density between osteophytes and normal bone, the better.

[0031] To ensure stable and consistent extraction of basic edge information, the weights of these Sobel kernels are fixed and set to be non-trainable. Compared to other edge detection algorithms, the Sobel operator is insensitive to image noise, making it well-suited for processing clinical medical images that often contain various artifacts.

[0032] Pooling Branch: This branch focuses on extracting macroscopic spatial layout information from the image. The spatial distribution and internal structure of abnormal bone spurs (osteophytes) differ from normal bone. Max-pooling downsamples the feature map by taking the maximum value within a local neighborhood. This not only reduces the feature dimensionality and computational cost, but more importantly, it enhances the model's robustness to geometric transformations such as translation. This allows the model to better understand the structural relationship between osteophytes and surrounding normal bone from a spatial context perspective.

[0033] Finally, the feature maps output from the two branches are fused at the end of the module. Specifically, the edge feature map extracted by the Sobel branch and the spatial structure feature map extracted by the pooling branch are concatenated along the channel dimension. This forms a more comprehensive low-level feature map that contains both high-frequency edge details and preserves low-frequency global structure, providing a rich and robust input for the subsequent dual-backbone network.

[0034] Steps S2 and S3: Differentiated dual backbone network and Stitch adaptive fusion.

[0035] To effectively address the significant differences in osteophyte size, morphology, and background trabecular bone texture among different patients, and to enhance the representation of fine-grained cortical bone texture, this invention constructs a differentiated dual-backbone parallel network architecture. The core principle of this design is "feature complementarity," meaning that the two backbone networks extract features from different but complementary perspectives, and these features are fused through a lightweight stitch fusion module (such as...). Figure 3 As shown, this enables adaptive and dynamic fusion of information.

[0036] Dual-backbone architecture:

[0037] Backbone 1 (ConvFormer + CGLU): This backbone focuses on capturing local fine textures and establishing long-range dependencies. ConvFormer is a hybrid structure of convolutional and Transformer architectures. This invention introduces the ConvFormer architecture, which combines the powerful local feature extraction capabilities of convolutional neural networks (CNNs) with the advantages of the Transformer architecture in modeling global contextual relationships. Furthermore, by integrating the Convolutional Gated Linear Unit (CGLU) mechanism, the model can dynamically recalibrate feature responses through data-dependent gating weights, thereby more accurately identifying and filtering subtle feature patterns related to osteophytes.

[0038] Backbone 2 (HGBlock + DWConv): This backbone focuses on associating and aggregating semantic information across multiple scales. The HGBlock module, through its hierarchical and grouped design, effectively extracts and associates features at different scales, which is crucial for identifying ossicles of varying sizes and shapes. Subsequently, depthwise separable convolution (DWConv) is applied to process these multi-scale features, significantly reducing computational cost and parameter count while efficiently completing the feature extraction task and improving the overall efficiency of the model.

[0039] Stitch Fusion Module: Effectively fusing information from two backbone networks with different architectures and potentially different feature distributions is crucial to model performance. To address this, this invention designs a Stitch fusion module, the architecture of which is as follows: Figure 3 As shown. Figure 3 This is a schematic diagram of the design architecture of the Stitch fusion module in an embodiment of the present invention. This module can dynamically allocate contribution weights between the two backbones at the instance level (i.e., for each input X-ray image), allowing the network to flexibly balance local details and global context. The workflow of the Stitch fusion module consists of the following three steps:

[0040] 1) Channel alignment: Two sets of 1×1 convolutions are used to map the features to a uniform number of channels C, completing the initial feature space alignment:

[0041] (5)

[0042] in, The original feature map output by the first backbone network (ConvFormer + CGLU) contains fine-grained local texture information of osteophytes; The original feature map output by the second backbone network (HGBlock + DWConv) contains cross-scale global semantic information; For 1×1 convolution operations, the number of feature channels is adjusted to achieve spatial alignment; , They are respectively , The feature maps aligned by 1×1 convolution have the same number of channels.

[0043] 2) Content-aware weight estimation: The aligned features are concatenated along the channel dimension and fed into a 3×3 convolution + sigmoid activation function combination to generate an adaptive weight tensor W:

[0044] (6)

[0045] W is then divided into:

[0046] (7)

[0047] In the above formula (6) express and In the channel-dimensional concatenation, W is a pixel-level adaptive weight tensor with dimensions C×H×W (C is the number of channels, H is the feature map height, and W is the feature map width). The closer the value is to 1, the greater the contribution of the corresponding feature to osteophyte detection; [0,1] C×H×WThe range and dimensional constraints for W are defined. W1 and W2 represent the pixel-level weights corresponding to the first and second backbone feature maps, respectively. Essentially, this is a lightweight implementation of a content attention-based mechanism.

[0048] 3) Weighted fusion and refinement: The weights are element-wise multiplied with the original features and then summed. Two learnable global scaling factors γ1 and γ2 (initialized to 0.5, denoted as X1_Weight and X2_Weight in the figure) are introduced, and then a 1×1 convolution is performed to obtain the final fusion result.

[0049] (8)

[0050] Y represents the final output feature map after adaptive weighted fusion and refinement by the Stitch fusion module.

[0051] The learnable γ1 and γ2 provide global calibration capabilities, allowing the network to fine-tune its overall preferences during training, enabling the model to output stably even when faced with abnormal shooting angles or extreme lesions.

[0052] Step S4: Multi-scale semantic fusion and detection.

[0053] The features fused by the dual backbone networks require further multi-scale integration and optimization to address the challenges of osteophyte feature signals being susceptible to noise interference, large differences in lesion scale, and blurred boundaries. This invention designs a multi-scale semantic fusion module comprising an OFPN module, a HipOsteo-Meta module, and an SPDConv module.

[0054] OFPN module: The Osteophyte-Feature Pyramid Network (OPN) is located at the "neck" of the model. It is responsible for receiving the multi-scale output of the dual backbone network and providing semantically consistent and discriminative feature representations for the final detection head. Figure 4 This is a schematic diagram of the osteophyte feature pyramid network (OFPN) architecture in an embodiment of the present invention. Figure 4 As shown, OFPN makes three targeted improvements on the basis of the classic Feature Pyramid Network (FPN):

[0055] 1. Lightweight Channel Alignment: 1×1 convolutions are used as adjustment convolutions to unify the channel dimensions of input features at each level with minimal computational cost, preparing for subsequent fusion operations. This is the original feature map of layer l after dual-core fusion, where l is the feature pyramid level index and L is the total number of pyramid layers (L=4 in this invention). Different l values ​​correspond to osteophyte features at different scales. for The feature map aligned by 1×1 convolution is calculated as follows:

[0056] (9)

[0057] 2. Attention-Guided Mechanism: A compressed excitation module is superimposed on the aligned feature maps at each scale. This module adaptively learns the importance of each channel and recalibrates the channel responses. This gives higher weights to semantic channels related to osteophytes, while suppressing channels dominated by highlight artifacts, thereby enhancing the network's ability to focus on real lesions.

[0058] 3. Bidirectional Feature Path Aggregation: Traditional FPNs mostly adopt a top-down unidirectional information flow, while OFPNs adopt a bidirectional feature path that combines top-down and bottom-up approaches. This allows for more comprehensive information interaction between feature maps at different levels, enabling the transmission of strong semantic information from higher levels to shallower levels, and also supplementing higher levels with high-resolution detail information from shallower levels.

[0059] Figure 5 Detailed schematic diagrams of the hip joint osteophyte (hereinafter also referred to as HipOsteo-Meta) module and the spatial-to-depth convolution (hereinafter also referred to as SPDConv) module according to embodiments of the present invention are shown; wherein, Figure 5 (A) in the diagram details the structure of the convolution gated linear unit (CGLU), which is a structure used to process specific data. Figure 5 (B) shows the HipOsteo-Meta module, an advanced metadata processing module that enables complex analysis and management of data; Figure 5 (C) in the text specifically describes the SPDConv operation, which is a spatial-to-depth convolution.

[0060] HipOsteo-Meta module: such as Figure 5 As shown, to further improve the model's ability to detect small lesions and its robustness in complex clinical environments, the following components were introduced:

[0061] HipOsteo-Meta module: such as Figure 5 As shown in (A), this module is based on the flexible MetaFormer architecture and integrates the aforementioned CGLU gating mechanism. Figure 5 (B) in the image shows the HipOsteo-Meta module, an advanced metadata processing module that enables complex analysis and management of data.

[0062] The combination of MetaFormer's structural advantages and the efficiency of separable convolutions enables the modeling of complex interactions between different spatial regions or feature channels. CGLU's dynamic recalibration capability further enhances the model's stability under image quality fluctuations and complex backgrounds, enabling it to stably separate key pathological features from interfering information. This can be formalized as follows:

[0063] (10)

[0064] Where X is the feature map of the input HipOsteo-Meta module, with dimensions C×H×W; , The learnable convolutional kernel weights are used to extract basic features and generate gated masks, respectively. , To create a learnable bias term, adjust the offset between the base feature and the gate mask; σ represents a two-dimensional convolution operation; σ is the Sigmoid activation function; ⊙ represents a pixel-wise multiplication operation; Y is the refined feature map after CGLU processing.

[0065] SPDConv module: such as Figure 5 As shown in (C), considering that early, minute osteophytes occupy only a few pixels in X-ray images, traditional strided convolution or pooling operations are prone to losing these sub-pixel-level key clues during downsampling. Therefore, this invention introduces Space-to-Depth Convolution (SPDConv). SPDConv works by "rearranging first, then convolution": it first rearranges the spatial information of the input feature map to the channel dimension, halving the spatial size of the feature map and increasing the number of channels, before performing standard non-strided convolution. This approach achieves downsampling while avoiding direct information loss, "packaging" sub-pixel-level edge and texture clues into the channel dimension, greatly improving the model's sensitivity to small targets. The input is the original feature map of SPDConv, with dimensions C×H×W; the sub-pixel rearrangement positions are controlled, and the rearrangement method is as follows:

[0066] (11)

[0067] 2i+δh and 2j+δw are the pixel values ​​of the rearranged output feature map (c, i, j are the channel, height, and width indices of the output feature map); 2i+δh and 2j+δw are the height and width indices of the original feature map; δh and δw are the spatial offsets, with values ​​ranging from {0,1}. It represents the value of a specific pixel selected based on an offset within a specific 2×2 block of the original input feature map.

[0068] Finally, the feature map optimized by the multi-scale semantic fusion module is fed into the detection head, which outputs the precise location bounding box of the hip osteophyte and the corresponding confidence score. Figure 6 This is a heatmap showing the region of interest (ROI) on an X-ray image for the BTP-StitchNet model in an embodiment of the present invention. Figure 6 As shown in the heatmap visualization results, the model proposed in this invention can accurately locate the osteophyte region, and its scope of interest also reasonably covers the bone texture around the lesion, demonstrating the model's profound understanding of pathological features. Experimental results show that the method proposed in this invention outperforms existing mainstream detection models in key indicators such as precision, recall, and mean precision, while maintaining a high inference speed, meeting the needs of clinical applications.

Claims

1. A method for detecting hip osteophytes by adaptive fusion of texture perception and multi-scale features, characterized in that, Includes the following steps: Obtain X-ray images of the hip joint; The bone texture feature extraction module extracts and enhances the features of the hip joint X-ray image. The bone texture feature extraction module adopts a parallel dual-branch structure. The first branch uses a fixed-weight edge detection operator to extract gradient features, and the second branch uses pooling operation to extract spatial structure features. The feature maps output by the two branches are then stitched together in the channel dimension to obtain a feature map that integrates edge details and spatial structure. The fused feature map is input into a dual backbone network. The first backbone network extracts local fine texture and long-range dependency features through a hybrid structure combining convolution and self-attention and integrating a gating mechanism. The second backbone network extracts and associates multi-scale semantic features through a hierarchical grouping structure and depthwise separable convolution. Here, long-range dependency features refer to the semantic associations and contextual relationships between pixels or regions that are far apart in the image captured by the model. The adaptive fusion module dynamically weights and fuses the features output by the dual backbone network. The adaptive fusion module first aligns the channels of the dual backbone output features, then generates pixel-level adaptive weights through a content-aware mechanism, and uses a learnable global scaling factor to calibrate and fuse the weighted features to obtain the fused features. The fused features are input into a multi-scale semantic fusion module, which includes an osteophyte feature pyramid network with bidirectional feature pathways. Feature integration and optimization are achieved by channel alignment, attention recalibration and cross-level information interaction of features at different scales. Based on the optimized features, osteophyte detection is performed, and the location and confidence level of the osteophytes are output.

2. The method according to claim 1, characterized in that, In the bone texture feature extraction module, the edge detection operator is the Sobel operator. It uses 3×3 convolution kernels in the horizontal and vertical directions to perform convolution operations on the input image and calculates the gradient magnitude to enhance the edge features of the bone spurs. The convolution kernel weights of the Sobel operator are fixed and cannot be trained to stably extract basic edge information.

3. The method according to claim 1, characterized in that, The hybrid structure in the first backbone network can simultaneously capture local features and global contextual relationships; the integrated gating mechanism dynamically recalibrates the feature response through data-dependent gating weights to identify and filter osteophyte-related feature patterns.

4. The method according to claim 1, characterized in that, The hierarchical grouping structure in the second backbone network can extract and associate features at different scales; the depthwise separable convolution used processes multi-scale features, completing feature extraction while reducing computational cost and parameter quantity.

5. The method according to claim 1, characterized in that, The adaptive fusion module uses 1×1 convolutions to map the feature maps output by the dual backbone network to a uniform number of channels for channel alignment; content-aware weight estimation uses 3×3 convolutions and a sigmoid activation function to generate an adaptive weight tensor. The weighted fusion unit introduces two learnable global scaling factors to calibrate the weighted features.

6. The method according to claim 1, characterized in that, The osteophyte feature pyramid network in the multi-scale semantic fusion module adds lightweight channel alignment operation, attention guidance mechanism and bidirectional feature path aggregation on the basis of the classic feature pyramid structure; the bidirectional feature path aggregation includes two information flows, one from top to bottom and one from bottom to top, to realize information interaction between feature maps of different levels.

7. The method according to claim 6, characterized in that, The lightweight channel alignment uses 1×1 convolution to adjust the channel dimension of the input features at each level; the attention guidance mechanism uses a compressed activation module to adaptively learn the importance of each channel and recalibrate the channel response; the bidirectional feature path aggregation transmits the semantic information of the deep feature map to the shallow layer and supplements the resolution detail information of the shallow feature map to the deep layer.

8. The method according to claim 1, characterized in that, The multi-scale semantic fusion module also includes a meta-feature extraction module, which is based on a meta-precursor structure and integrates a gating mechanism to enhance the model's ability to detect small lesions and its robustness in complex clinical environments.

9. The method according to claim 1, characterized in that, The multi-scale semantic fusion module also includes a space-to-depth convolution module, which downsamples the input feature map by rearranging the spatial information to the channel dimension, thus avoiding the loss of sub-pixel-level edge and texture cues.

10. The method according to claim 1, characterized in that, The osteophyte detection step specifically involves: inputting the feature map optimized by the multi-scale semantic fusion module into the detection head, outputting the bounding box position and confidence score of the osteophyte, and completing the end-to-end detection process.

Images