A method and system for bone age prediction based on a multi-modal large model and a computer readable storage medium
By fusing left-hand DR images with clinical indicators through an end-to-end multimodal large model, a description of bone-by-bone imaging findings and maturity level are generated, which solves the accuracy and efficiency problems of bone age assessment in existing technologies and realizes the automation and compliance of bone age prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHAN DONG MSUN HEALTH TECH GRP CO LTD
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-05
AI Technical Summary
Existing bone age assessment methods rely on single-modal image processing, ignoring objective clinical indicators such as actual age, current height, and gender. This leads to a disconnect between bone age prediction and actual developmental status. Furthermore, two-stage cascading is prone to introducing errors, and the output lacks descriptions of bone-by-bone imaging findings. Doctors face a heavy workload in manually writing reports, and purely generative reports may produce inaccurate or contradictory descriptions.
An end-to-end multimodal large model is used to fuse left-hand DR images with multimodal objective indicators such as actual age, current height, and gender. Through a hybrid visual backbone, a bone age-specific local attention enhancement module, a clinical indicator-guided projector, and a bone-by-bone grading classification head, a description of bone-by-bone images and maturity level are generated, and the compliance of bone age calculation is ensured through rule constraints.
It enables the automatic generation of descriptions of bone-by-bone imaging findings and maturity levels, improving the accuracy of bone age prediction and the efficiency of clinical assessment, reducing the burden on doctors to manually write reports, and ensuring the compliance and interpretability of the output.
Smart Images
Figure CN122158083A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal bioinformatics processing, and more specifically to a method, system, and computer-readable storage medium for predicting bone age based on a large multimodal model. Background Technology
[0002] Bone age assessment is a core indicator for pediatric growth and development evaluation, endocrine disease diagnosis, and adult height prediction. Currently, clinical practice primarily relies on physicians manually comparing the assessment to the Chinese Wrist Bone Development Standard 05 (《Chinese Wrist Bone Development Standard - Chinese 05》) or the TW3 / GP atlas. This involves scoring each of the 20 major bones in the left wrist (distal radius, distal ulna, carpal bones, metacarpals, phalanges, etc.) for ossification centers, epiphyseal width exceeding diaphysis, and epiphyseal-diaphysis fusion, and then calculating the total bone age value using a weighted scoring table. This process requires a comprehensive assessment of the child's actual age, current height, gender, and parents' height to determine whether bone age is advanced, lagging, or normal, and to predict adult height.
[0003] The Chinese Standard 05 is a revision based on the TW3 scoring method and is highly applicable to Chinese children, especially for abnormal cases such as short stature and precocious puberty. However, manual interpretation suffers from problems such as strong subjectivity, poor consistency, and long processing time, making it difficult to complete efficiently, particularly in primary healthcare institutions.
[0004] In recent years, artificial intelligence technology has made progress in bone age assessment. Existing methods are mainly divided into two categories. The first is single-modal image processing, which uses CNN or Transformer to directly regress the total bone age value from the left-hand DR image, or calculates it after bone segmentation / keypoint detection. This type of method has a high degree of automation, but ignores objective clinical indicators such as actual age, current height, and gender, resulting in the model's inability to effectively learn the correlation between bone age and actual developmental status. The second is a two-stage or cross-modal architecture, such as the "Method and Device for Generating Bone Age Reports" disclosed in CN117912627A, which first outputs structured results through a bone age detection model, and then uses an independent report generation model to fill a preset template to generate a report; another example is the "Two-Stage Bone Age Assessment System Based on Cross-Modal Learning for Left-Hand X-ray Images" disclosed in CN121236501A, which combines the fixed text description of the standard bone age atlas for cross-modal alignment, first performing coarse-grained interval classification, and then fine-grained regression of the total bone age value.
[0005] While the aforementioned existing technologies have improved the level of automation to some extent, they still have the following defects and shortcomings: Existing methods largely rely solely on image features, leading to a disconnect between bone age prediction and actual developmental status. They fail to fully utilize objective clinical indicators such as actual age, current height, and gender, and cannot accurately capture patterns of advanced / lagging bone age. Two-stage cascaded methods are prone to error propagation and have low inference efficiency; the output is often a single total bone age value, lacking descriptions of each bone image, placing a heavy burden on doctors to manually write reports. Purely generative report models may produce inaccurate or contradictory descriptions, easily leading to misinterpretations; bone age calculations often rely on model regression and do not strictly embed the Chinese 05 weighted scoring rules, resulting in potential biases.
[0006] Therefore, there is an urgent need for an end-to-end method that integrates left-hand DR images with multimodal objective indicators such as actual age, current height, and gender to achieve automatic generation of bone-by-bone image descriptions and maturity levels, and to ensure compliance of bone age calculations through rule constraints, so as to truly reduce the burden of doctors manually writing reports and improve overall clinical efficiency and accuracy. Summary of the Invention
[0007] The purpose of this invention is to provide a method, system, and storage medium for bone age prediction based on a multimodal large model, which enables the synchronous output of natural language descriptions and maturity levels for each bone image, thereby improving the accuracy of prediction, the interpretability of results, and the completeness of clinical assessment.
[0008] To achieve the above objectives, the present invention employs the following technical solutions.
[0009] A bone age prediction method based on a multimodal large model, characterized by the following steps: S1. Obtain the left-hand DR image and clinical indicators of the subject to be predicted; S2. The left hand DR image and the structured prompts carrying the clinical indicators are used as inputs to a pre-trained multimodal large model. The multimodal large model outputs the image descriptions of each bone in the left wrist and the bone-by-bone structured results of maturity level. S3. Based on the maturity level in the bone-by-bone structuring results, calculate the total bone age value according to the scoring rules of the preset bone age assessment standard. S4. Generate and output a bone age assessment report that includes a description of the bone-by-bone images, maturity level, and total bone age value.
[0010] Furthermore, the clinical indicators include at least actual age, current height, and gender. After obtaining the clinical indicators, the actual age, current height, and gender are standardized respectively. After obtaining the left-hand DR image, it is standardized, the pixel values are mapped to the range of [0,255], converted into a single-channel grayscale image, normalized to the range of [0,1], and adjusted to the resolution supported by the multimodal large model. The steps for standardizing left-hand DR images are as follows: If the image contains window width Window position information Then, standardization is performed according to the following formula: , , , in, These are the original pixel values. The pixel values after window processing. To round down; If the image does not carry window width and window level information, then the histogram of all pixel values in the image is calculated, and the lower and upper limits of the dynamic window width are selected to map the original pixel values to the [0,255] interval.
[0011] Furthermore, the structured prompts include fixed-domain instructions for bone age assessment and output format requirements for bone-by-bone structured results, with the clinical indicators embedded as prefixes in the structured prompts.
[0012] Furthermore, the multimodal large model is a visual-language large model that has been customized and fine-tuned in the bone age domain, including a hybrid visual backbone, a bone age-specific local attention enhancement module, a clinical indicator-guided projector, and a bone-by-bone grade classification head; during model training, the pre-trained visual backbone and most language model parameters are frozen, and only the parameters of newly added modules are fine-tuned; The internal processing flow of a multimodal large model includes: The input image is fed into the visual tower, and a hybrid visual backbone model is used to extract image features, which include global context features and local multi-scale detail features. Global average pooling is performed on global context features and local multi-scale detail features to obtain global vectors. Bone age region bias weights are calculated and local feature enhancement is performed on image features. The enhanced feature maps are then converted into visual token sequences through a patch bedding layer.
[0013] Furthermore, the multimodal large model employs a multi-task constrained loss function. train: , The multi-task constraint loss function includes generation loss, bone-by-bone level classification loss, and rule-guided consistency constraint loss. The total loss is the weighted sum of these losses, where... , , For weight hyperparameters, Autoregressive cross-entropy loss is used to ensure the natural coherence of the description seen in bone-by-bone images; For bone-by-bone classification loss, a level-weighted cross-entropy method is used, with an imbalanced distribution of level samples. To guide the consistency loss, the mapping rule of "image performance → level" is embedded in the loss calculation, which forces the generated descriptive text to maintain semantic consistency with the output maturity level.
[0014] Furthermore, the consistency constraint loss generates a pseudo-rank distribution by extracting keywords from the image-seen description, calculates the KL divergence between the pseudo-rank distribution and the maturity rank distribution directly output by the multimodal large model, and applies a penalty term to cases of contradiction between description and rank, thereby achieving semantic consistency between image-seen description and maturity rank. The calculation formula is as follows: , in, It is a pseudo-rank distribution. This is the maturity level distribution directly output by the multimodal large model. For adjustable hyperparameters, For the first The rule for violating the skeleton is penalized.
[0015] Furthermore, the preset bone age assessment standard is the Chinese 05 standard, and the scoring rule is the RUS-CHN scoring method. The weight and maturity level of each bone are obtained through a hard-coded mapping table. After weighted summation to obtain the total score, the total bone age value is calculated by interpolation with the preset bone age reference curve.
[0016] Furthermore, the bone-by-bone structured results are parsed using regular expressions or a string parser to extract the image descriptions and maturity levels of each bone before the calculation steps of the scoring rules are executed.
[0017] A bone age prediction system based on a multimodal large model includes: The data acquisition module is used to acquire the left hand DR image of the subject to be predicted, as well as clinical indicators such as actual age, current height, and gender. A prompt building module is used to construct structured prompts carrying the aforementioned clinical indicators; The model inference module has a built-in pre-trained multimodal large model, which is used to receive the left-hand DR image and structured prompts, and output bone-by-bone structured results; The bone age calculation module is used to parse the bone-by-bone structured results and calculate the total bone age value according to the scoring rules of the preset bone age assessment standard. The report output module is used to generate and output bone age assessment reports.
[0018] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described bone age prediction method based on a multimodal large model.
[0019] The advantages of this invention are: By adopting an end-to-end multimodal fusion architecture, the left-hand DR image is input into the model along with clinical objective indicators such as actual age, height, and gender. This establishes a deep correlation between bone age and the actual developmental status of children, which can accurately capture developmental patterns of advanced and lagging bone age and significantly improve the accuracy of bone age prediction. By directly outputting the natural language description and maturity level of the images of each bone in the left wrist through a multimodal large model, fine-grained bone-by-bone structured results are automatically generated, completely eliminating the heavy burden of doctors manually writing detailed bone age reports and greatly improving the efficiency of clinical bone age assessment. A rule-guided consistency constraint loss function was designed to effectively eliminate the illusion and contradictory description problems of pure generative models. At the same time, the weighted scoring rules of the Chinese 05 standard were hard-coded into an independent calculation module, strictly following clinical standards to calculate bone age values, avoiding the calculation bias of pure model regression, and ensuring the clinical compliance and reliability of the model output. Lightweight customization and fine-tuning of the pre-trained visual language model in the bone age domain is performed. The core parameters are frozen and only the newly added functional modules are fine-tuned. While retaining the model's general visual language understanding ability, the computational cost of model training is greatly reduced, making it easy to be engineered and implemented in dedicated software in hospital systems or primary healthcare institutions. The system architecture features a modular design, integrating data processing, model inference, bone age calculation, and report output. It is easy to operate and adaptable to real-world clinical applications. Attached Figure Description
[0020] Figure 1 This is a flowchart of the bone age prediction method based on a multimodal large model according to the present invention. Detailed Implementation
[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0022] Example 1 This embodiment discloses a bone age prediction method based on a multimodal large model. This method can be applied to computers in hospital systems or dedicated bone age assessment software. For specific steps, please refer to... Figure 1 The process includes the following steps.
[0023] Step S101: Image preprocessing stage The preprocessing stage mainly includes standardization and normalization of the original left-handed DR image to adapt to the input requirements of a multimodal large language model. If the DICOM data tag carries window width (WW) and window level (WL) information, standardization is performed according to the window width and window level, linearly mapping the pixel values to the range [0, 255], converting it into a single-channel grayscale image. The processing formula is: , , , in, These are the original pixel values. The pixel values after window processing. This is for rounding down.
[0024] If the DICOM tag does not carry window width and window level information, then a histogram of all pixel values in the image is plotted, and the 0.5% and 99.5% percentile values P_0.005 and P_0.995 are selected as the lower and upper limits of the dynamic window width, respectively, for linear mapping. , After processing, the images are uniformly adjusted to the resolution supported by the model (e.g., 448×448 pixels) and normalized to the [0,1] interval.
[0025] .
[0026] Step S102: Construction of Patient Information and Structured Prompts Obtain the child's objective clinical indicators: actual age A, current height H (cm), and gender G. The actual age A is recorded in the format of "X years Y months" and converted to months a = 12×X + Y.
[0027] Construct structured prompt text P as a dialog prefix for the model: P = “Patient Information: Actual Age” + str(a) + “Months, Current Height” + str(H) + “cm, Gender” + G + “ Based on the Chinese 05 Bone Age Assessment Standard, the imaging features of 20 skeletal regions (distal radius, distal ulna, carpal bones, metacarpals, proximal phalanges, middle phalanges, and distal phalanges) in the left hand X-ray were analyzed, and the bone-by-bone structured results were output in the following format: "Bone Name: Description of Imaging Findings; Grade: X" The prompt embeds clinical indicators as prefixes to guide the model in learning the relationship between bone age and actual age / height.
[0028] Step S103: Forward Inference of Multimodal Large Language Model The preprocessed image I input The input text P is used together with the multimodal large language model BoneAge-MLLM.
[0029] The internal processing flow of the model is as follows: The first step involves inputting the image into the visual tower, where a custom hybrid visual backbone model is used to extract image features, and ViT-L / 16 is used to extract global contextual features. Swin-Transformer-T extracts local multi-scale detail features. Secondly, a bone age-specific local attention enhancement module (BoneAge-Local Attention Booster, BLAB) is introduced, primarily targeting... and Global average pooling is performed separately to obtain the global vector. and Then, the bone age region bias weights are calculated. Finally, the bone age region bias weights are used to... and Perform local feature enhancement, and then enhance the 2D feature map. Through the patch embedding layer (to...) Flattening (from patch sequence and linear projection) into a visual token sequence Where R represents the real number field, This represents the number of visual tokens, and d=1024 represents the feature dimension of each token.
[0030] The formula for calculating the bone age region bias weight is as follows: , Where H and W represent the height and width of the feature map, respectively, concat represents the concatenation operation, MLP represents a multilayer perceptron, and the output dimension is R. H×W×1 sigmoid represents the sigmoid activation function, which maps the values output by the MLP to the interval [0,1].
[0031] The formula for calculating enhanced local features is as follows: , The second step is for the text tokenizer to convert P into a text token sequence T.
[0032] The third step involves inputting clinical indicators, such as age and height, into the clinical indicator-guided projector to combine visual feature information with clinical information. The clinical indicator-guided projector mainly consists of three modules: a 3-layer MLP, dynamic numerical injection, and bone age deviation perception attention. First, age, height, and gender are concatenated as individual clinical features. Second, clinical indicators are dynamically injected, and then the resulting visual token sequence is processed. Input the bone age deviation perception attention module to calculate the global visual average vector. By fusing clinical indicators and global visual information through query vectors, we obtain... Finally, clinically guided visual features and visual token input standard multi-head attention mechanisms will be incorporated to adjust the visual features for biased perception, resulting in adjusted fused visual features. Further, the fused visual features Concatenate the input with the text token sequence T, and input the Qwen2.5 language model to generate the output sequence via autoregression.
[0033] The formula for embedding clinical indicators is as follows: , , , , in, , , These are the embedding vectors for age, height, and gender features, respectively. Embedding is a learnable embedding layer, one-hot encoding is used, and the formula for dynamically injected parameters is: , Where pos_enc is the learnable positional encoding. For layer normalization.
[0034] The formula for calculating the global visual average vector is: , That is, the average of all tokens in V_enh is used to obtain a global representation of the entire image.
[0035] The formula for calculating the fusion of clinical indicators and global visual information is as follows: The formula for calculating the fused visual feature h_vis_guided is: MultiHeadAttention is a standard multi-head attention mechanism. Q comes from clinical guidance, and K / V comes from visual tokens, enabling the adjustment of visual features for bias perception.
[0036] The output is structured text, for example: Distal radius: The medial end of the epiphysis is the same width as the shaft; Grade: 6 Distal ulna: The epiphyseal styloid process is visible as a small, well-defined protrusion. Grade: 4 ... (20 complete skeletons).
[0037] Step S104: Output Analysis and Bone Age Calculation (Chinese 05) Extract the description of each bone from the output sequence using regular expressions or a string parser. and level ( ).
[0038] Calculate the total score S according to the RUS-CHN scoring rules for China 05: in The weights are determined by a preset fixed table, such as w=1.0 for the distal radius, w=0.9 for the distal ulna, and w=0.8 for the first metacarpal bone. The corresponding score for each grade can be obtained by referring to the scoring table for the 2005 Chinese Football Association (CFA) program. =0 gets 0 points, =10, which is 178 points.
[0039] Bone age (BA) was obtained by referring to a pre-defined control curve based on the total score (S). in( , )and( , The two points in the comparison table are the closest points. Both the comparison curve and the comparison table refer to the Chinese Standard 05.
[0040] Step S105: Report Generation and Output Describing bone by bone ,grade The total bone age (BA) is integrated into a structured report text and output to the display interface or PDF file.
[0041] The finely tuned multimodal large language model BoneAge-MLLM is obtained through the following steps: Step S1061: Building the BoneAge-MLLM module A multimodal large language model for bone age prediction, BoneAge-MLLM, was constructed with Qwen2-VL-7B-Instruct as its core architecture. This model is deeply customized for the bone age domain based on Qwen2-VL. Key innovations include the integration and optimization of the following components to ensure the model fully understands image context, clinical indicators (age, height, gender), and the Chinese 05 rule prior, generating high-quality bone-by-bone structured descriptions and grading.
[0042] A custom hybrid vision backbone is used for visual feature extraction, employing a fusion architecture that combines global context (ViT-L / 16) and local multi-scale details (Swin-Transformer-T). Then, a bone age-specific local attention booster (BLAB) module is introduced to adaptively focus on the skeletal regions of the left-hand DR image. The BLAB module structure is based on the feature maps output by ViT and Swin. and Global average pooling is performed separately to obtain the global vector. and By calculating the bias weights of bone age regions, local features are further enhanced, resulting in an enhanced 2D feature map. The sequence is converted into a visual token sequence through a patch embedding layer (flattened into a patch sequence + linear projection). (d=1024).
[0043] The Clinical-Guided Projector consists of three modules: a 3-layer MLP, dynamic numerical injection, and bone age deviation perception attention.
[0044] Step S1062: Construction of a large-scale bone age instruction dataset The bone age dataset contains a large number of image-text pairs. Its main purpose is to take left-hand DR images and patient information as input to BoneAge-MLLM and output bone-by-bone structured descriptions and grades. The dataset images and annotations are from collaborating hospitals, including provincial tertiary hospitals and municipal hospitals, and the bone-by-bone grading was performed by radiologists (conforming to the Chinese O5 standard).
[0045] The instruction design uniformly adopts a structured prompt template consistent with the reasoning phase to ensure alignment between training and reasoning: answer (enter command): "Patient Information: Actual Age" + str(a_sample) + "months, Current Height" + str(H_sample) + "cm, Gender" + G_sample + "". Based on the Chinese 2005 Bone Age Assessment Standard, the bones in the left hand X-ray were analyzed, and the bone-by-bone structured results were output in the following format: "Bone Name: Description of Imaging Findings; Grade: X" response (target output, labeled by the physician): "Distal radius: The medial end of the epiphysis is the same width as the shaft; Grade: 6" Distal ulna: The epiphyseal styloid process is visible as a small, distinct bulge; Grade: 5 First metacarpal bone: Epiphyses cover the shaft on both sides. ; Grade: 8 ...(20 complete skeletons, in a fixed order) Step S1063: BoneAge-MLLM Training and Fine-tuning Phase Training uses LoRA fine-tuning and is conducted in stages: Phase 1: Freeze the hybrid visual backbone (including the BLAB module) and most of the Qwen2.5 language model parameters, and only fine-tune the clinical indicator-guided projector and LoRA adapter layer to quickly adapt to the clinical indicator injection and bone-by-bone structure generation tasks.
[0046] The second stage involves unfreezing the hybrid visual backbone layer by layer (first unfreezing the Swin local feature layer, then unfreezing the ViT global layer). After unfreezing each layer, the accuracy of bone-level extraction is evaluated on the validation set to optimize the sensitivity of visual feature extraction to bone age images.
[0047] Phase 3: Jointly fine-tune the visual backbone and language model, reduce the learning rate to 1e-5, and enhance cross-modal fusion capabilities.
[0048] The model training aims at autoregressive language modeling, adjusting parameters through backpropagation: in, Output sequence for target (bone-by-bone structured description + rank). For sequence length, Let be the token at position t in the sequence. This refers to all tokens generated up to the t-th token.
[0049] To simultaneously ensure the coherence of the generated text, the accuracy of the bone-by-bone grading, and the consistency of the rules governing the description and grading, this invention employs a multi-task constrained total loss function: in , , The weight hyperparameter (based on experience and multiple experiments, the possible values are:) , , ), The constraint can be gradually increased in the later stages of training to strengthen consistency.
[0050] (Autoregressive cross-entropy loss): Applied to the output head of the language model to ensure that the generated text is fluent and natural. (Bone-by-bone classification loss, weighted cross-entropy): Applied to the bone-by-bone classification head attached to the model, it extracts features from the intermediate layers or final hidden states of the language model to predict the classification category of each bone, ensuring accurate classification prediction. Where: C is the number of level categories, for example, C=12 corresponds to levels 0-12. For the true one-hot label of the i-th bone, These are the predicted probabilities output by the softmax model. Weights represents the prior frequency of class c, used to alleviate the sparsity problem of low / high-level samples in the China 05 dataset.
[0051] L_consist (rule-guided consistency loss, the core innovation of this invention): operates on the post-processing supervision module after generation or during the generation process, extracting keywords from the descriptive text generated by the model. A pseudo-level distribution is generated based on the preset mapping table of China 05. , and the model's classification head prediction Calculate the KL divergence and impose penalties for contradictory cases to ensure that the description is consistent with hierarchical semantics and conforms to clinical rules: The KL divergence is defined as: The penalty item is defined as: in, For the i-th keyword extracted from the generated text, such as "complete epiphyseal fusion", The level predicted by the model for the corresponding skeleton can be the highest probability level. According to the pre-defined grade requirements for this keyword in the Chinese Standard 05, such as "complete epiphyseal fusion", the corresponding... =14, It is a hyperparameter that can be dynamically adjusted based on the description-level contradictions on the validation set.
[0052] Example 2 The bone age prediction system based on a multimodal large model in this invention includes a data acquisition module, a preprocessing module, a prompt construction module, a model inference module, a bone age calculation module, and a report output module, corresponding to the above-mentioned method. Each module is a hardware or software module, integrated into a hospital's PACS system or a dedicated bone age assessment device. The modules work together to achieve automated and intelligent prediction of bone age. The specific module functions are as follows: Data acquisition module: It is equipped with an image reading interface and a clinical information input interface. It can automatically retrieve the left hand DR image of the subject to be predicted from the hospital PACS system, and also supports the import of local DR image files. At the same time, it supports manual input or synchronization of the clinical indicators of the subject to be predicted from the hospital electronic medical record system, including actual age, current height, and gender information, to complete the raw data collection and unified storage for bone age prediction. The preprocessing module includes built-in image standardization and normalization algorithms and clinical indicator format conversion logic. It performs window processing, resolution unification, and pixel value normalization operations on the left-hand DR images acquired by the data acquisition module to obtain image data that meets the input requirements of a multimodal large model. At the same time, it converts the actual age into months format and standardizes the encoding of height and gender information to provide standardized clinical indicator data for subsequent prompt construction. The prompt construction module: It pre-stores fixed domain instruction templates for bone age assessment, and can automatically retrieve standardized clinical indicators output by the preprocessing module. It then concatenates these indicators according to a preset format to generate structured prompt text carrying clinical indicators. This text includes patient information prefixes, Chinese 05 standard assessment instructions, and bone-by-bone structured result output format requirements to ensure that it matches the input requirements of the model inference module. Model Inference Module: Built-in multimodal large model BoneAge-MLLM, which is customized and fine-tuned in the bone age domain. This model integrates core components such as hybrid visual backbone, bone age-specific local attention enhancement module BLAB, and clinical indicator guided projector. It can receive preprocessed left hand DR images and structured prompt text generated by the prompt construction module. Through multimodal feature fusion, autoregressive generation and other processes, it outputs bone-by-bone structured results containing descriptions of the bones seen in the left wrist images and maturity levels. Bone age calculation module: It has a built-in regular expression parser, a hard-coded mapping table of RUS-CHN scoring rules of the Chinese 05 standard, and a bone age interpolation calculation algorithm. First, it parses the bone-by-bone structured results output by the model inference module and extracts the maturity level of each bone. Then, it obtains the weight and corresponding score of each bone level according to the mapping table, and calculates the total score by weighted summation. Finally, it queries the discretized reference table of bone age reference curve through the total score, selects adjacent reference points and performs linear interpolation calculation to obtain the total bone age value. Report output module: It pre-stores clinical report templates for bone age assessment, and can automatically integrate the total bone age value output by the bone age calculation module, the description of bone-by-bone imaging findings and maturity level output by the model inference module, to generate a structured bone age assessment report that conforms to clinical standards. It supports the visualization of the report on the terminal display interface, and also provides report export functions in commonly used formats such as PDF and Word. It can also support the synchronization of reports to the hospital's electronic medical record system to achieve interconnection and interoperability of clinical data.
[0053] Example 3 The computer-readable storage medium in this embodiment of the invention is a non-volatile storage medium, including but not limited to solid-state drives (SSDs), portable hard drives, USB flash drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, etc., which stores computer programs. The computer programs are executable instruction sets. When the instruction sets are called and executed by the processor, they can realize all the steps of the bone age prediction method based on a multimodal large model in the above embodiments.
[0054] Specifically, when the computer program is executed by the processor, it performs the following operations in sequence: Step S1: Call the data reading interface to obtain the left hand DR image of the subject to be predicted, as well as clinical indicators such as actual age, current height, and gender; Step S2: Perform standardization and normalization preprocessing on the left hand DR image, and convert the clinical indicators into a preset format. Standardization is achieved by window processing or percentile method depending on whether the DR image carries window width WW and window level WL information. Normalization adjusts the image to the resolution supported by the model and maps the pixel values to the [0,1] interval. Step S3: Construct structured cue text based on the converted clinical indicators, and input the preprocessed left hand DR image and structured cue text into a multimodal large model that has been customized and fine-tuned in the bone age domain; Step S4: Image features are extracted through the hybrid visual backbone of a multimodal large model. After the bone region features are enhanced by a bone age-specific local attention enhancement module, the projector is guided by clinical indicators to achieve multimodal feature fusion. Finally, bone-by-bone structured results are generated through language model autoregression. Step S5: Analyze the bone-by-bone structured results, extract the maturity level of each bone, calculate the total score according to the RUS-CHN scoring rules of the Chinese 05 standard, and then obtain the total bone age value through linear interpolation. Step S6: Generate a bone age assessment report that includes a description of bone-by-bone imaging findings, maturity level, and total bone age value, and display or export the report through the output interface.
[0055] The computer program can be executed by different processors and is compatible with various hardware devices such as ordinary computers, medical terminals, bone age assessment all-in-one machines, and hospital PACS system servers. When executed on an ordinary computer, it can realize offline processing of bone age prediction, which is suitable for primary medical institutions without dedicated assessment equipment. When executed on a medical terminal or PACS system server, it can realize online real-time processing of bone age prediction, which meets the efficient diagnostic needs of hospital clinical outpatient departments.
[0056] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A bone age prediction method based on a multimodal large model, characterized in that, Including the following steps: S1. Obtain the left-hand DR image and clinical indicators of the subject to be predicted; S2. The left hand DR image and the structured prompts carrying the clinical indicators are used as inputs to a pre-trained multimodal large model. The multimodal large model outputs the image descriptions of each bone in the left wrist and the bone-by-bone structured results of maturity level. S3. Based on the maturity level in the bone-by-bone structuring results, calculate the total bone age value according to the scoring rules of the preset bone age assessment standard. S4. Generate and output a bone age assessment report that includes a description of the bone-by-bone images, maturity level, and total bone age value.
2. The bone age prediction method based on a multimodal large model according to claim 1, characterized in that, The clinical indicators include at least actual age, current height, and gender. After obtaining the clinical indicators, the actual age, current height, and gender are standardized respectively. After acquiring the left-hand DR image, it is standardized to map the pixel values to the range of [0,255], converted into a single-channel grayscale image, normalized to the range of [0,1], and adjusted to the resolution supported by the multimodal large model; The steps for standardizing left-hand DR images are as follows: If the image contains window width Window position information Then, standardization is performed according to the following formula: ， ， ， in, These are the original pixel values. The pixel values after window processing. To round down; If the image does not carry window width and window level information, then the histogram of all pixel values in the image is calculated, and the lower and upper limits of the dynamic window width are selected to map the original pixel values to the [0,255] interval.
3. The bone age prediction method based on a multimodal large model according to claim 1, characterized in that, The structured prompts include fixed-domain instructions for bone age assessment and output format requirements for bone-by-bone structured results, with the clinical indicators embedded as prefixes in the structured prompts.
4. The bone age prediction method based on a multimodal large model according to claim 1, characterized in that, The multimodal large model is a visual-language large model customized and fine-tuned in the bone age domain, including a hybrid visual backbone, a bone age-specific local attention enhancement module, a clinical indicator-guided projector, and a bone-by-bone grade classification head; during model training, the pre-trained visual backbone and most language model parameters are frozen, and only the parameters of newly added modules are fine-tuned; The internal processing flow of a multimodal large model includes: The input image is fed into the visual tower, and a hybrid visual backbone model is used to extract image features, which include global context features and local multi-scale detail features. Global average pooling is performed on global context features and local multi-scale detail features to obtain global vectors. Bone age region bias weights are calculated and local feature enhancement is performed on image features. The enhanced feature maps are then converted into visual token sequences through a patch bedding layer.
5. The bone age prediction method based on a multimodal large model according to claim 4, characterized in that, The multimodal large model employs a multi-task constraint loss function. train: ， The multi-task constraint loss function includes generation loss, bone-by-bone level classification loss, and rule-guided consistency constraint loss. The total loss is the weighted sum of these losses, where... , , For weight hyperparameters, Autoregressive cross-entropy loss is used to ensure the natural coherence of the description seen in bone-by-bone images; For bone-by-bone classification loss, a level-weighted cross-entropy method is used, with an imbalanced distribution of level samples. To guide the consistency loss, the mapping rule of "image performance → level" is embedded in the loss calculation, which forces the generated descriptive text to maintain semantic consistency with the output maturity level.
6. The bone age prediction method based on a multimodal large model according to claim 5, characterized in that, The consistency constraint loss generates a pseudo-rank distribution by extracting keywords from the image-seen description, calculates the KL divergence between the pseudo-rank distribution and the maturity rank distribution directly output by the multimodal large model, and applies a penalty term to cases of contradiction between description and rank, thereby achieving semantic consistency between image-seen description and maturity rank. The calculation formula is as follows: ， in, It is a pseudo-rank distribution. This is the maturity level distribution directly output by the multimodal large model. For adjustable hyperparameters, For the first The rule for violating the skeleton is penalized.
7. The bone age prediction method based on a multimodal large model according to claim 1, characterized in that, The preset bone age assessment standard is the Chinese 05 standard, and the scoring rule is the RUS-CHN scoring method. The weight and maturity level of each bone are obtained through a hard-coded mapping table. After weighted summation to obtain the total score, the total bone age value is calculated by interpolation with the preset bone age reference curve.
8. The bone age prediction method based on a multimodal large model according to claim 1, characterized in that, The bone-by-bone structured results are parsed using regular expressions or a string parser to extract the image descriptions and maturity levels of each bone, and then the calculation steps of the scoring rules are executed.
9. A bone age prediction system based on a multimodal large model, characterized in that, include: The data acquisition module is used to acquire the left hand DR image of the subject to be predicted, as well as clinical indicators such as actual age, current height, and gender. A prompt building module is used to construct structured prompts carrying the aforementioned clinical indicators; The model inference module has a built-in pre-trained multimodal large model, which is used to receive the left-hand DR image and structured prompts, and output bone-by-bone structured results. The bone age calculation module is used to parse the bone-by-bone structured results and calculate the total bone age value according to the scoring rules of the preset bone age assessment standard. The report output module is used to generate and output bone age assessment reports.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the bone age prediction method based on a multimodal large model as described in any one of claims 1-8.