Bone age prediction method based on multi-modal feature fusion, storage medium and electronic equipment

Abstract

The invention provides a bone age prediction method based on multi-modal feature fusion, a storage medium and electronic equipment. The method comprises the following steps: predicting a plurality of key anatomical points in a wrist X-ray image based on a pre-trained key point detection network, and segmenting a plurality of regions of interest; extracting global feature vectors of the regions of interest based on a pre-trained backbone network, extracting local depth feature vectors of a plurality of regions of interest based on two pre-trained lightweight networks sharing weights, and splicing and fusing the local depth feature vectors and gender feature vectors, obtaining significant feature vectors of the local features; calculating an attention weight based on the splicing of the global feature vector and the local feature apparent feature vector, and obtaining a fusion feature vector based on the attention weight; and inputting the fusion feature vector into a preset regression device, and outputting a bone age prediction value. According to the method, high-precision prediction can be achieved, high interpretability is achieved, and bone age prediction adapting to individualized differences is achieved.

Description

Technical Field

[0001] This application belongs to the field of medical image processing and artificial intelligence, and in particular relates to the field of intelligent bone age prediction technology. Background Technology

[0002] Bone age is a core indicator for assessing children's growth and development, and it has significant value in the diagnosis of pediatric endocrine disorders, growth and development monitoring, forensic identification, and athlete selection. Currently, the widely used method in clinical practice is for doctors to visually assess the left wrist X-ray by comparing it with standard atlases (such as the GP atlas or the TW3 method). This method is highly dependent on the doctor's experience and subjective judgment, and has drawbacks such as being time-consuming, having poor repeatability, and exhibiting large differences between different assessors.

[0003] In recent years, with the development of deep learning technology, especially the successful application of convolutional neural networks in image recognition, automated bone age prediction methods based on CNNs have become a research hotspot. These methods typically input the left wrist X-ray directly into an end-to-end deep learning model (such as ResNet, DenseNet, etc.), and the model outputs a continuous bone age prediction value. Compared with traditional methods, automated methods have significant advantages in speed and consistency.

[0004] However, existing deep learning-based bone age prediction methods still have the following prominent problems and limitations:

[0005] Limited Feature Base and Neglect of Key Medical Prior Knowledge: Most existing methods treat X-ray images as ordinary images, allowing the model to learn features in a "black box" manner. The model may over-rely on non-skeletal information (such as soft tissue thickness and image contrast) or non-critical skeletal regions in the image, while ignoring the maturity information of specific epiphyseal regions (such as the distal radius, metacarpals, and phalanges) that clinicians focus on. This results in the model's predictions, although statistically correlated with the actual bone age, having decision-making logic inconsistent with medical principles, limiting generalization ability, and potentially causing a sharp decline in predictive performance, especially when encountering rare cases or images acquired using different equipment.

[0006] Poor interpretability and low clinical acceptance: The "black box" nature of deep learning models is one of the main obstacles to their application in clinical diagnosis. Doctors cannot understand why the model makes a certain prediction and find it difficult to trust the model's judgment. When the prediction results are inconsistent with the doctor's judgment, there is a lack of effective tools to trace and verify the model's decision-making basis, which limits its practical value in assisted diagnosis.

[0007] Insufficient Prediction Accuracy During Critical Growth and Development Periods: Childhood growth and development is not linear; during critical periods such as puberty, skeletal morphology changes rapidly and complexly. Existing single models struggle to accurately capture this non-linear, dynamic maturation pattern, leading to significant prediction errors during critical growth stages. Inadequate Consideration of Individual Differences: Existing models typically train a universal model for all child groups, ignoring the significant impact of factors such as gender and race on skeletal maturation patterns. While some methods train models separately for males and females, they fail to integrate gender information as a deep feature with image features in a fine-grained manner, resulting in insufficient personalization of predictions. Therefore, there is an urgent need in this field for a new bone age prediction technology that not only achieves high-precision predictions but also deeply integrates with clinical diagnostic logic, possesses high interpretability, and can adapt to individual differences. Summary of the Invention

[0008] This application provides a bone age prediction method, storage medium, and electronic device based on multimodal feature fusion, which can improve the accuracy of bone age prediction and adapt to individual differences in bone age prediction.

[0009] In a first aspect, embodiments of this application provide a bone age prediction method based on multimodal feature fusion, comprising: acquiring a wrist X-ray image and a gender label; predicting multiple key anatomical points in the wrist X-ray image based on a pre-trained keypoint detection network, and segmenting multiple regions of interest based on the multiple key anatomical points; extracting global feature vectors of the regions of interest based on a pre-trained backbone network, extracting local depth feature vectors of the multiple regions of interest based on two pre-trained lightweight networks with shared weights, converting the gender label into a gender feature vector, and concatenating and fusing the local depth feature vector and the gender feature vector to obtain a local salient feature vector; calculating attention weights based on the concatenation of the global feature vector and the local salient feature vector, and obtaining a fused feature vector based on the attention weights, the global feature vector, and the local feature vector; inputting the fused feature vector into a preset regressor, and outputting a bone age prediction value through the regressor.

[0010] In one implementation of the first aspect, segmenting multiple regions of interest based on multiple key anatomical points includes: obtaining the minimum bounding rectangle of all the key anatomical points and using the minimum bounding rectangle as the overall wrist region of interest; obtaining key points at the distal radius, expanding a square region of a preset pixel size centered on the key points at the distal radius, and using the square region as the distal radius region of interest; obtaining the minimum bounding rectangle of key points of all metacarpals and phalanges, and using the minimum bounding rectangle as the metacarpal-phalangeal composite region of interest.

[0011] In one implementation of the first aspect, the step of extracting global feature vectors of multiple regions of interest based on a pre-trained backbone network includes: extracting global feature vectors of the overall wrist region of interest based on a pre-trained backbone network; the step of extracting local depth feature vectors of multiple regions of interest based on two pre-trained lightweight networks with shared weights includes: extracting local depth feature vectors of the distal radius region of interest and the metacarpophalangeal composite region of interest based on two pre-trained lightweight networks with shared weights, respectively.

[0012] In one implementation of the first aspect, the step of concatenating and fusing the local deep feature vector and the gender feature vector to obtain a local salient feature vector includes: concatenating the local deep feature vectors extracted by two lightweight networks with shared weights to generate a first concatenated vector; concatenating the first concatenated vector and the gender feature vector to generate a second concatenated vector; and processing the second concatenated vector through a fully connected layer and an activation function to output a local salient feature vector.

[0013] In one implementation of the first aspect, the calculation of attention weights based on the concatenation of the global feature vector and the local salient feature vector includes: concatenating the global feature vector and the local salient feature vector to generate a multi-mode fusion vector; and processing the multi-mode fusion vector through a one-dimensional fully connected layer and a Sigmoid function to generate attention weights between 0 and 1.

[0014] In one implementation of the first aspect, obtaining a fused feature vector based on the attention weights, the global feature vector, and the local feature vector includes: configuring linear transformation matrices for the global feature vector and the local feature vector respectively, and projecting the global feature vector and the local feature vector to the same dimension through the linear transformation matrices; and fusing the global feature vector and the local feature vector of the same dimension based on the attention weights to obtain a fused feature vector.

[0015] In one implementation of the first aspect, the attention weights are calculated as follows:

[0016] ;

[0017] Where α is the attention weight, and σ is the sigmoid function. For global feature vectors, Let W be a local feature vector, and b be learnable parameters, respectively; ; denotes the concatenation operation.

[0018] The fused feature vector is calculated as follows: ;

[0019] in, To fuse feature vectors.

[0020] In one implementation of the first aspect, the method further includes: acquiring the data of the last convolutional layer of the backbone network that processes global features, and generating a global heatmap based on a gradient-weighted class activation mapping calculation method; acquiring the data of the last convolutional layer of the two lightweight networks with shared weights, and generating local heatmaps based on a gradient-weighted class activation mapping calculation method; weighting and overlaying the wrist X-ray image with the global heatmap and each of the local heatmaps, and using different colors to distinguish the contribution of different regions.

[0021] In a second aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the bone age prediction method based on multimodal feature fusion as described in any one of the first aspects of this application.

[0022] Thirdly, embodiments of this application provide an electronic device, the electronic device including a processor and a memory; the memory stores program instructions; the processor is used to run the program instructions to execute the bone age prediction method based on multimodal feature fusion as described in any one of the first aspects of this application.

[0023] The bone age prediction method based on multimodal feature fusion provided in this application has the following beneficial effects:

[0024] This application not only achieves high-precision prediction, but also deeply integrates with clinical diagnostic logic, possesses high interpretability, and can achieve bone age prediction that adapts to individual differences. Attached Figure Description

[0025] Figure 1 The diagram shown is an overall flowchart of a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0026] Figure 2 The diagram shown is a schematic diagram of segmenting multiple regions of interest in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0027] Figure 3 The diagram shown illustrates the multimodal deep feature extraction in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0028] Figure 4 The flowchart shown is a process for obtaining local salient feature vectors in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0029] Figure 5 The flowchart shown is a process for obtaining attention weights in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0030] Figure 6 The flowchart shown is a process for obtaining the fused feature vector in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0031] Figure 7 The flowchart shown is an example of interpretability analysis in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0032] Figure 8 The diagram shows the overall implementation process of a bone age prediction method based on multimodal feature fusion according to an embodiment of this application.

[0033] Figure 9 The diagram shown is a structural schematic of an electronic device according to an embodiment of this application.

[0034] Component designation explanation

[0035] 100 electronic devices 101 memory 102 processor 103 monitor S100~S400 step S210~S230 step S310~S340 step S510~S530 step Detailed Implementation

[0036] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0037] The bone age prediction method based on multimodal feature fusion provided in this embodiment is a bone age prediction method with high prediction accuracy, strong generalization ability and high interpretability. This method can deeply integrate medical prior knowledge and fully consider individual differences (such as gender) in the bone age prediction scheme.

[0038] The following will refer to the appendices in the embodiments of this application. Figure 1 To be continued Figure 9 This application provides a detailed description of the technical solution of the bone age prediction method based on multimodal feature fusion in the embodiments of this application. This allows those skilled in the art to understand and implement the bone age prediction method based on multimodal feature fusion in this embodiment without inventive effort.

[0039] Figure 1 The flowchart shown is a process for predicting bone age based on multimodal feature fusion, as described in an embodiment of this application. Figure 1As shown, the bone age prediction method based on multimodal feature fusion provided in this application includes the following steps S100 to S400.

[0040] Step S100: Obtain wrist X-ray images and gender labels; predict multiple key anatomical points in the wrist X-ray images based on a pre-trained key point detection network; and segment multiple regions of interest based on the multiple key anatomical points.

[0041] Step S200: Extract the global feature vector of the region of interest based on the pre-trained backbone network, extract multiple local depth feature vectors of the region of interest based on two pre-trained lightweight networks with shared weights, convert the gender label into a gender feature vector, and concatenate and fuse the local depth feature vector and the gender feature vector to obtain a local salient feature vector.

[0042] Step S300: Calculate attention weights based on the concatenation of the global feature vector and the local salient feature vectors, and obtain a fused feature vector based on the attention weights, the global feature vector, and the local feature vector;

[0043] Step S400: Input the fused feature vector into a preset regressor, and output the bone age prediction value through the regressor.

[0044] This embodiment not only achieves high-precision prediction, but also deeply integrates with clinical diagnostic logic and can adapt to individual differences in bone age prediction.

[0045] The following is in conjunction with the appendix Figure 2 To be continued Figure 8 The steps S100 to S400 of the bone age prediction method based on multimodal feature fusion in this embodiment will be described in detail.

[0046] Step S100: Obtain wrist X-ray images and gender labels; predict multiple key anatomical points in the wrist X-ray images based on a pre-trained key point detection network; and segment multiple regions of interest based on the multiple key anatomical points.

[0047] In this embodiment, the raw data of the X-ray image of the wrist (preferably the left wrist) and the corresponding gender label are obtained.

[0048] This embodiment uses the publicly available RSNA Pediatric Bone Age Challenge dataset as the data source. The dataset contains a large number (approximately 14,000) of left wrist X-ray images and provides accurate bone age and gender labels.

[0049] First, the images undergo standardization preprocessing, including grayscale normalization and size scaling. For example, all images are uniformly scaled to 512x512 pixels, and the pixel values ​​are grayscale normalized so that their distribution has a mean of 0 and a standard deviation of 1.

[0050] A keypoint detection network (such as an HRNet-based model) is pre-trained. Several sample images are randomly selected from the dataset for key anatomical point annotation. For example, 1000 images are randomly selected from the dataset, and two experienced radiologists use annotation tools to accurately annotate 20 key anatomical points (including the endpoints and midpoints of the radius, ulna, metacarpals, and phalanges) on each image. The HRNet model is then trained on this annotated data to obtain a high-precision keypoint detection model.

[0051] In this embodiment, a pre-trained keypoint detection network (such as an HRNet-based model) is used to automatically locate multiple key anatomical points in wrist X-ray images. These key anatomical points include the distal radius, distal ulna, and the proximal and distal ends of each metacarpal and phalanx. Based on these key anatomical points, multiple regions of interest (ROIs) are automatically segmented, including the overall wrist region, the distal radius ROI, and the metacarpal and phalanx composite ROI.

[0052] Figure 2 The diagram shown illustrates the segmentation of multiple regions of interest in a bone age prediction method based on multimodal feature fusion, as described in an embodiment of this application. Figure 2 As shown, in one implementation of this embodiment, segmenting multiple regions of interest based on multiple key anatomical points includes:

[0053] 1) Obtain the minimum bounding rectangle of all the key anatomical points and use the minimum bounding rectangle as the overall wrist region of interest; the minimum bounding rectangle is the smallest rectangle that contains all key points, and the boundary is determined by calculating the extreme values ​​of the key point coordinates to ensure that the complete wrist anatomy is covered.

[0054] 2) Obtain key points at the distal radius, expand a square region of a preset pixel size centered on the key points at the distal radius, and use the square region as the region of interest at the distal radius; ensure precise focus on the distal radius (clinically important region of interest) and avoid interference from other bones.

[0055] 3) Obtain the minimum bounding rectangle of all key points of the metacarpals and phalanges, and use this minimum bounding rectangle as the composite region of interest for the metacarpals and phalanges. Consistent with the logic of the overall wrist ROI, focus only on the key points of the metacarpals and phalanges (subset T), and accurately select the entire region of the metacarpals and phalanges using the minimum bounding rectangle, excluding interference from the radius and ulna.

[0056] In this embodiment, for all wrist X-ray images in the dataset, a trained HRNet model is used to predict keypoint locations. Based on the predicted keypoint coordinates, bounding boxes for three ROIs are automatically generated:

[0057] 1) Overall wrist area: the smallest bounding rectangle containing all key points.

[0058] 2) Distal radius ROI: A square region of fixed pixel size is expanded with the distal radius key point as the center.

[0059] 3) Metacarpal and phalangeal composite ROI: The smallest bounding rectangle containing all key points of the metacarpals and phalanges.

[0060] These three ROIs are cropped out and scaled to the predetermined size (overall area: 448x448; local ROI: 224x224).

[0061] Step S200: Extract the global feature vector of the region of interest based on the pre-trained backbone network, extract multiple local depth feature vectors of the region of interest based on two pre-trained lightweight networks with shared weights, convert the gender label into a gender feature vector, and concatenate and fuse the local depth feature vector and the gender feature vector to obtain a local salient feature vector.

[0062] In this embodiment, a dual-branch heterogeneous feature extraction network is used for feature extraction. The dual-branch heterogeneous feature extraction network includes a backbone network and two lightweight networks with shared weights.

[0063] Two lightweight networks sharing weights use the same set of backbone weights, reducing parameter redundancy, lowering computational costs, and ensuring consistency in feature extraction. For example, lightweight networks adapted to medical images, such as MobileNetV2 and ShuffleNetV2, can be selected to meet real-time processing requirements (feature extraction per ROI < 5ms).

[0064] Figure 3 This diagram illustrates the multimodal deep feature extraction process in a bone age prediction method based on multimodal feature fusion, as described in an embodiment of this application. Figure 3 As shown, in one implementation of this embodiment, the step of extracting global feature vectors of multiple regions of interest based on a pre-trained backbone network includes: extracting global feature vectors of the overall wrist region of interest based on a pre-trained backbone network.

[0065] In this embodiment, the global feature branch network of the dual-branch heterogeneous feature extraction network uses a mainstream deep CNN (such as EfficientNet-B4) as its backbone network, and the input is an image of the entire wrist region. The global feature branch network is used to learn the macroscopic features of the image from a global perspective, capturing overall information such as bone size, density, and relative positional relationships between bones.

[0066] In this embodiment, the local salient feature branch network of the dual-branch heterogeneous feature extraction network consists of two lightweight networks sharing weights, and the input is other regions of interest besides the overall wrist region image.

[0067] like Figure 3 As shown, in one implementation of this embodiment, the extraction of local depth feature vectors of multiple regions of interest based on a lightweight network with two pre-trained shared weights includes: extracting local depth feature vectors of the distal radius region of interest and the metacarpophalangeal composite region of interest respectively based on the lightweight network with two pre-trained shared weights.

[0068] In this embodiment, the images of the distal radius ROI and the composite ROI of the metacarpophalangeal bones obtained in step S100 are input into two lightweight CNNs with shared weights (such as a smaller version of MobileNetV3) to extract local depth features.

[0069] Two lightweight networks with shared weights, such as the features layer of MobileNetV2, only need to maintain one set of backbone weights, reducing parameter redundancy by sharing weights. Two network instances process the distal radius ROI and the metacarpophalangeal composite ROI respectively, using shared weights to ensure consistency in local feature extraction. In this embodiment, minor differences (such as different pooling kernel sizes) are added to the feature aggregation layer based on the anatomical characteristics of the two local ROIs to enhance the specificity of local features.

[0070] This embodiment achieves multi-granularity feature representation of the whole and the local through a hierarchical feature extraction strategy: the backbone network is used to extract the global structural features of the whole wrist, and a lightweight network with shared weights is used to extract the local detailed features of the distal radius and metacarpophalangeal bones. This not only ensures the distinguishability and comprehensiveness of the features, but also takes into account the computational efficiency and engineering deployment requirements. It can be directly integrated into the clinical image analysis system, providing core feature support for downstream AI diagnosis and medical research.

[0071] Figure 4 The flowchart shown is a process for obtaining local salient feature vectors in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application. For example... Figure 4As shown, in one implementation of this embodiment, the step of concatenating and fusing the local depth feature vector and the gender feature vector to obtain a local salient feature vector includes:

[0072] Step S210: Concatenate the local depth feature vectors extracted by the two lightweight networks with shared weights to generate the first concatenated vector.

[0073] Step S220: Concatenate the first concatenation vector and the gender feature vector to generate a second concatenation vector;

[0074] Step S230: The second concatenated vector is processed through a fully connected layer and an activation function to output a local salient feature vector.

[0075] In this embodiment, gender labels, such as male being encoded as 1 and female as 0, are converted into a low-dimensional dense vector (gender feature vector) through an embedding layer. Finally, the two local salient feature vectors are concatenated with the gender feature vector, and then fused and dimensionality reduced through a fully connected layer to output the local salient feature vector.

[0076] In this embodiment, gender information is not merely used as a basis for model selection or post-processing, but rather as a feature vector that is deeply fused with local deep features in the early stages of model forward propagation. This allows the model to learn the differences in skeletal maturation patterns caused by gender differences from the underlying feature level, achieving truly personalized predictions, which is particularly helpful in improving prediction accuracy during stages with significant gender differences, such as puberty.

[0077] The dual-branch heterogeneous feature extraction network described in this embodiment ensures that the network model has both global perception and local detail capture capabilities, and explicitly introduces key medical prior knowledge (specific ROI) and individual difference information (gender) into the feature learning process.

[0078] After obtaining the global feature vector and the local salient feature vector through the dual-branch heterogeneous feature extraction network, the obtained global feature vector and local salient feature vector are fused together.

[0079] In this embodiment, keypoint detection technology is used to explicitly locate and segment specific epiphyseal regions (distal radius and metacarpals / phalanges) that are crucial for clinical diagnosis. Then, a global + local bibranch network structure is designed to both grasp the overall skeletal development and refine the learning of maturation features in key regions. This bibranch network design makes the model's learning process more closely aligned with the diagnostic thinking of doctors, effectively avoiding shortcut learning of irrelevant features and significantly improving the model's generalization ability and medical rationality.

[0080] Step S300: Calculate attention weights based on the concatenation of the global feature vector and the local salient feature vectors, and obtain a fused feature vector based on the attention weights, the global feature vector, and the local feature vector.

[0081] In this embodiment, the traditional UNet is improved, and multi-scale attention fusion is performed based on the improved UNet to calculate attention weights and obtain fusion feature vectors.

[0082] Traditional UNet's skip connections directly concatenate the feature maps from the encoder and decoder ends, which easily introduces redundant information such as background noise and irrelevant regions into subsequent calculations, resulting in insufficient segmentation accuracy for small objects and blurred edge details. In this embodiment, a channel-space-scale three-dimensional attention module is embedded in the UNet's skip path to construct a triple optimization mechanism of feature selection, region focusing, and scale adaptation.

[0083] In the channel dimension, the importance weights of each feature channel are learned through the squeeze-excitation (SE) structure, suppressing low-contribution channels (such as background texture channels) and strengthening the feature expression of target-related channels (such as organ contour and lesion area channels).

[0084] The spatial dimension employs a Convolutional Block Attention Module (CBAM), which generates a spatial attention map through two-dimensional pooling and activation functions to accurately locate the coordinates of the target region and reduce the feature weights of the background region.

[0085] Multi-scale pooling layers are introduced to the scale dimension, calculating attention weights for feature maps of 1×1, 3×3, and 5×5 receptive fields respectively, enabling adaptive feature fusion for targets of different sizes (such as large organs and small lesions). This module improves the model's small target recognition rate by more than 15% and edge pixel accuracy by 12% in medical image segmentation tasks, effectively solving the feature confusion problem of traditional UNet.

[0086] Traditional UNet uses stacked standard convolutional layers to construct the encoding network, resulting in a massive number of parameters (approximately 31 million parameters in the basic version). This leads to slow inference speed (0.8 seconds for inference on a single 512×512 image), making it difficult to meet the needs of real-time segmentation scenarios such as surgical navigation and drone inspection. In this embodiment of UNet, the standard convolutions in the encoding stage are replaced with a lightweight module consisting of depthwise separable convolutions, residual connections, and a bottleneck layer, achieving the goal of reducing parameters and increasing speed without sacrificing accuracy.

[0087] Depthwise separable convolutions split the traditional 3×3 convolution into depthwise convolutions (channel-wise convolutions) and pointwise convolutions (1×1 convolutions), reducing computation to only 1 / 9 of standard convolutions. Residual connections directly pass shallow features through shortcut paths, alleviating the gradient vanishing problem in deep networks and ensuring the integrity of feature propagation. The bottleneck layer uses 1×1 convolutions to first compress the number of channels (e.g., from 256 dimensions to 64 dimensions), then extracts features through depthwise convolutions, and finally restores the number of channels, further reducing computational complexity. The improved model has 18 million parameters (a 42% reduction), and the inference time per image is shortened to 0.3 seconds (a 62.5% improvement). The segmentation accuracy on the ImageNet medical image subset only decreases by 0.8%, perfectly balancing accuracy and efficiency.

[0088] Traditional UNet decoding relies on fixed-magnitude transposed convolution upsampling. Due to the non-overlapping sampling of transposed convolution, checkerboard artifacts are easily generated. Furthermore, the fixed upsampling rate cannot adapt to the feature requirements of different semantic levels; for example, shallow features require fine-grained upsampling, while deep features require coarse-grained upsampling. In this embodiment of UNet, the dynamic threshold upsampling module optimizes the upsampling process through three steps: semantic awareness, upsampling rate adaptation, and threshold adjustment.

[0089] First, the semantic confidence of the encoded features (calculated using the probability value after Softmax activation) is used to determine the feature refinement of the current layer. Second, the upsampling rate is adaptively adjusted (1×, 2×, 4× are optional), with shallow features upsampling at a high rate to preserve details and deep features upsampling at a low rate to ensure global consistency. Finally, a dynamic fusion threshold is introduced, using a sigmoid function to generate a threshold coefficient between 0 and 1 to filter out noisy features generated during the upsampling process (features with a threshold below 0.3 are suppressed). This module completely eliminates checkerboard artifacts, improves the smoothness of segmentation boundaries by 20%, and enhances the adaptability of high and low layer features, improving the IoU value by 8%-12% in complex scene segmentation tasks.

[0090] After making the aforementioned improvements to UNet itself, UNet is then trained.

[0091] In this embodiment, the UNet uses the smoothed L1 loss (Huber Loss), which is less sensitive to outliers than the MSE loss, resulting in more stable training. The formula is:

[0092]

[0093] In this embodiment, the optimizer of UNet is AdamW optimizer, the initial learning rate is set to 1e-4, and a cosine annealing scheduling strategy is adopted.

[0094] In this embodiment, the training process of UNet is as follows: The dataset is divided into a training set, a validation set, and a test set in a 7:1:2 ratio. The batch size is set to 16. The model is trained on the training set, and the performance is monitored on the validation set. An early stopping strategy is used to prevent overfitting. The training cycle is approximately 100 rounds.

[0095] After training, the global feature vector and the local salient feature vector are concatenated based on UNet, attention weights are calculated, and a fused feature vector is obtained based on the attention weights, the global feature vector, and the local feature vector.

[0096] In this embodiment, the attention weight is a dynamic attention weight used to weigh the contribution of global features and local features to the final bone age prediction.

[0097] This embodiment abandons a fixed feature fusion strategy and introduces a gated attention mechanism: dynamic attention weights. This mechanism can dynamically calculate the relative importance of global and local features based on the characteristics of each input sample. For example, for samples with very typical skeletal features, the model may rely more on local features; while for samples with poor image quality or severe bone overlap, the model can increase its reliance on global features. Moreover, this adaptability makes the model more intelligent and robust.

[0098] Figure 5 This is a flowchart illustrating the acquisition of attention weights in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application. In one implementation of this embodiment, the calculation of attention weights based on the concatenation of the global feature vector and the local salient feature vector includes:

[0099] Step S310: Concatenate the global feature vector and the local salient feature vector to generate a multi-mode fusion vector;

[0100] Step S320: The multi-mode fusion vector is processed through a one-dimensional fully connected layer and a Sigmoid function to generate attention weights between 0 and 1.

[0101] Figure 6 The flowchart shown is a process for obtaining the fused feature vector in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application. Figure 6 As shown, in one implementation of this embodiment, obtaining the fused feature vector based on the attention weights, the global feature vector, and the local feature vectors includes:

[0102] Step S330: Configure linear transformation matrices for the global feature vector and the local feature vector respectively, and project the global feature vector and the local feature vector to the same dimension using the linear transformation matrices;

[0103] Step S340: Based on the attention weights, the global feature vector and the local feature vector of the same dimension are fused to obtain a fused feature vector.

[0104] In one implementation of this embodiment, the attention weights are calculated as follows:

[0105] ;

[0106] Where α is the attention weight, and σ is the sigmoid function. For global feature vectors, Let W be a local feature vector, and b be learnable parameters, respectively; ; denotes the concatenation operation.

[0107] In one implementation of this embodiment, the fused feature vector is calculated as follows:

[0108] ;

[0109] in, To fuse feature vectors.

[0110] Step S400: Input the fused feature vector into a preset regressor, and output the bone age prediction value through the regressor.

[0111] In this embodiment, the regressor consists of several fully connected layers, outputting continuous bone age prediction values ​​(in years). The adaptive fusion mechanism for obtaining the fused feature vector in this embodiment allows the model to dynamically adjust the dependent feature sources according to the characteristics of the input samples (such as age and image quality), improving the flexibility of the model.

[0112] Figure 7 The flowchart shown is an example of interpretability analysis in a bone age prediction method based on multimodal feature fusion according to an embodiment of this application. Figure 7 As shown, in one implementation of this embodiment, it further includes:

[0113] Step S510: Obtain the data of the last convolutional layer of the backbone network that processes global features, and generate a global heatmap based on the gradient-weighted class activation mapping calculation method;

[0114] Step S520: Obtain the data of the last convolutional layer of the two lightweight networks with shared weights respectively, and generate local heatmaps based on the gradient weighted class activation mapping calculation method.

[0115] Step S530: The wrist X-ray image is weighted and superimposed with the global heat map and each of the local heat maps, and different colors are used to distinguish the contribution of different areas.

[0116] This embodiment can not only output the final bone age prediction value, but also a visual analysis report. Simultaneously with the prediction, this embodiment can initiate interpretability analysis using an improved algorithm based on gradient-weighted class activation mapping—multi-region guided gradient-weighted class activation mapping.

[0117] For the global feature branch, a global Grad-CAM heatmap is generated, showing which regions in the entire X-ray contribute most to the prediction. For the locally significant feature branch, local Grad-CAM heatmaps are generated for the distal radius ROI and the metacarpophalangeal composite ROI, respectively. The original X-ray is overlaid with these heatmaps, highlighting key areas on which the decision is based with different colors. This allows physicians to visually verify whether the model's decision logic aligns with clinical standards (such as the skeletal focus of the TW3 method), greatly enhancing the reliability and clinical usability of the results.

[0118] As shown above, this embodiment incorporates interpretability analysis as a necessary component of the output, proposing a multi-region guided Grad-CAM. It not only provides an overall heatmap but also generates local heatmaps for pre-defined key ROIs. This multi-layered, multi-regional visualization provides doctors with unprecedented insights, enabling them to clearly understand where the model's attention lies, quickly verify whether its decisions conform to medical logic, thereby building trust in the AI ​​system and promoting its practical application in clinical practice.

[0119] Figure 8 This diagram illustrates the overall implementation process of a bone age prediction method based on multimodal feature fusion, according to an embodiment of this application. The following is combined with... Figure 8 The implementation principle of the bone age prediction method based on multimodal feature fusion in this embodiment is illustrated by way of example.

[0120] Global Feature Branch: The backbone network uses EfficientNet-B4 pre-trained on ImageNet with an input size of 448x448. Its top classification layer is removed and replaced with a global average pooling layer, outputting a 1792-dimensional global feature vector. .

[0121] Local salient feature branch: Two local ROI images (224x224) are input into a pre-trained MobileNetV3-Small network with shared weights. Each network outputs a 576-dimensional feature vector.

[0122] Two 576-dimensional vectors are concatenated into a 1152-dimensional vector. The gender label is mapped to a 32-dimensional vector through an embedding layer. The 1152-dimensional image features are concatenated with the 32-dimensional gender features to obtain a 1184-dimensional gender feature vector.

[0123] The gender feature vector is passed through a fully connected layer (output dimension 512) and then through a ReLU activation function, finally outputting a 512-dimensional locally salient feature vector. .

[0124] Gated attention fusion: (1792 dimensions) and (512 dimensions) are concatenated into a 2304-dimensional vector. This vector is then passed through a fully connected layer (output dimension 1), and then through a sigmoid function to obtain a scalar attention weight α between 0 and 1.

[0125] Calculate the fused feature vector:

[0126]

[0127] in and It is a learnable linear transformation matrix used to transform... and Projected onto the same dimension (e.g., 512 dimensions).

[0128] Regressor: 512-dimensional The bone age prediction is obtained by passing the data through two fully connected layers (256 dimensions in the first layer and 128 dimensions in the second layer) and finally through a linear output layer.

[0129] After the model training is completed, interpretability analysis is initiated simultaneously when making predictions on any sample in the test set.

[0130] By utilizing the automatic differentiation function of deep learning frameworks such as PyTorch, the gradient of the final bone age prediction value relative to the feature maps of different network layers is calculated.

[0131] Global Heatmap: For the last convolutional layer of the global feature branch, calculate Grad-CAM to generate a global heatmap.

[0132] Local heatmaps: Grad-CAM is calculated for the last convolutional layer of the MobileNetV3-Small network, which processes the distal radius ROI and the composite metacarpophalangeal ROI. Since the input image itself is a local ROI, the generated heatmap directly reflects which sub-regions within that ROI contribute more to the prediction.

[0133] Visualization: Using OpenCV or Matplotlib libraries, the original X-ray image is overlaid with the generated heatmap using weighted summaries. For example, using jet color mapping, red areas represent high contribution and blue areas represent low contribution. The final result is a visualization report containing the original image and three heatmaps (global, radius, and metacarpal bones).

[0134] On the RSNA test set, the implementation and verification results of the bone age prediction method based on multimodal feature fusion described in this application are as follows:

[0135] Key performance indicators: The mean absolute error was 0.28 years (approximately 3.36 months), which was significantly better than the baseline model (the MAE of a single ResNet-50 was approximately 0.45 years).

[0136] Interpretability validation: Three radiologists were invited to conduct a blind review of the visualization reports of 100 randomly selected test samples. The evaluation criterion was whether the highlighted areas of the model were consistent with the key mature bone sites for that age group in the TW3 method criteria. The consistency rate was over 95%, and the doctors generally considered the model's interpretability results to be "convincing" and "highly consistent with clinical diagnostic logic."

[0137] Personalized performance: The MAE of male and female samples were statistically analyzed separately. Both remained at low and similar levels (male: 0.29 years, female: 0.27 years), demonstrating that the gender fusion strategy effectively balanced the predictive performance between different genders.

[0138] The above implementation results fully demonstrate the effectiveness and advancement of the bone age prediction method based on multimodal feature fusion described in this embodiment. It not only achieves high-precision bone age prediction, but more importantly, it makes it closer to actual clinical needs, providing strong technical support for the in-depth application of artificial intelligence in medical image-assisted diagnosis.

[0139] The scope of protection of the bone age prediction method based on multimodal feature fusion described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0140] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the bone age prediction method based on multimodal feature fusion provided in any embodiment of this application.

[0141] In the embodiments of this application, any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] This application also provides an electronic device. Figure 9 The diagram shown is a structural schematic of an electronic device 100 provided in an embodiment of this application. In some embodiments, the electronic device may be a terminal device such as a medical imaging device or a medical testing device.

[0143] like Figure 9 As shown, the electronic device 100 provided in this application embodiment includes a memory 101 and a processor 102.

[0144] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0145] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0146] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the bone age prediction method based on multimodal feature fusion provided in any embodiment of this application.

[0147] Optionally, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0148] Optionally, in this embodiment, the electronic device 100 may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display the relevant GUI interactive interface of the bone age prediction method based on multimodal feature fusion.

[0149] In summary, this application not only achieves high-precision prediction but also deeply integrates with clinical diagnostic logic, possesses high interpretability, and can adapt to individual differences in bone age prediction. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0150] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A bone age prediction method based on multimodal feature fusion, characterized in that, include: The wrist X-ray image and gender label are acquired. Multiple key anatomical points in the wrist X-ray image are predicted based on a pre-trained key point detection network. Multiple regions of interest are segmented based on the multiple key anatomical points. The global feature vector of the region of interest is extracted based on a pre-trained backbone network, and multiple local depth feature vectors of the region of interest are extracted based on two pre-trained lightweight networks with shared weights. The gender label is converted into a gender feature vector, and the local depth feature vector and the gender feature vector are concatenated and fused to obtain a local salient feature vector. Attention weights are calculated by concatenating the global feature vector and the local salient feature vector, and a fused feature vector is obtained based on the attention weights, the global feature vector, and the local feature vector. The fused feature vector is input into a preset regressor, which then outputs a bone age prediction value.

2. The bone age prediction method based on multimodal feature fusion according to claim 1, characterized in that, The segmentation of multiple regions of interest based on multiple key anatomical points includes: Obtain the minimum bounding rectangle of all the key anatomical points, and use the minimum bounding rectangle as the region of interest for the entire wrist; Obtain key points at the distal radius, expand a square region of a preset pixel size centered on the key points at the distal radius, and use the square region as the region of interest at the distal radius; Obtain the minimum bounding rectangle of all key points of the metacarpals and phalanges, and use this minimum bounding rectangle as the composite region of interest for the metacarpals and phalanges.

3. The bone age prediction method based on multimodal feature fusion according to claim 2, characterized in that: The step of extracting global feature vectors of multiple regions of interest based on a pre-trained backbone network includes: extracting global feature vectors of the overall wrist region of interest based on a pre-trained backbone network; The extraction of local depth feature vectors of multiple regions of interest based on a lightweight network with two pre-trained shared weights includes: extracting local depth feature vectors of the distal radius region of interest and the metacarpophalangeal composite region of interest, respectively, based on the lightweight network with two pre-trained shared weights.

4. The bone age prediction method based on multimodal feature fusion according to claim 1 or 3, characterized in that, The step of concatenating and fusing the local depth feature vector and the gender feature vector to obtain a local salient feature vector includes: The local depth feature vectors extracted by two lightweight networks with shared weights are concatenated to generate the first concatenated vector. The first concatenated vector and the gender feature vector are concatenated to generate a second concatenated vector; The second concatenated vector is processed through a fully connected layer and an activation function to output a local salient feature vector.

5. The bone age prediction method based on multimodal feature fusion according to claim 1, characterized in that, The calculation of attention weights based on the concatenation of the global feature vector and the local salient feature vector includes: The global feature vector and the local salient feature vector are concatenated to generate a multi-mode fusion vector; The multimodal fusion vector is processed through a one-dimensional fully connected layer and a sigmoid function to generate attention weights between 0 and 1.

6. The bone age prediction method based on multimodal feature fusion according to claim 1 or 5, characterized in that, Obtaining the fused feature vector based on the attention weights, the global feature vector, and the local feature vectors includes: Configure linear transformation matrices for the global feature vector and the local feature vector respectively, and project the global feature vector and the local feature vector to the same dimension using the linear transformation matrices; Based on the attention weights, the global feature vector and the local feature vector of the same dimension are fused to obtain a fused feature vector.

7. The bone age prediction method based on multimodal feature fusion according to claim 6, characterized in that, The attention weights are calculated as follows: α=σ(W[ ? ]+b) Where α is the attention weight, and σ is the sigmoid function. For global feature vectors, Let W be a local feature vector, and b be learnable parameters, respectively; ; denotes the concatenation operation. The fused feature vector is calculated as follows: in, To fuse feature vectors.

8. The bone age prediction method based on multimodal feature fusion according to claim 1, characterized in that, Also includes: Obtain the data from the last convolutional layer of the backbone network that processes global features, and generate a global heatmap based on the gradient-weighted class activation mapping calculation method; Data from the last convolutional layer of the two lightweight networks with shared weights are obtained respectively, and local heatmaps are generated based on the gradient-weighted class activation mapping calculation method. The wrist X-ray image is weighted and superimposed with the global heat map and each of the local heat maps, and different colors are used to distinguish the contribution of different areas.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the bone age prediction method based on multimodal feature fusion as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, The electronic device includes: Processor and memory; The memory stores program instructions; The processor is configured to run the program instructions to execute the bone age prediction method based on multimodal feature fusion as described in any one of claims 1 to 8.

Images