A method for dynamic assessment of callus growth during fracture healing based on multimodal data.

Abstract

This invention relates to the field of artificial intelligence technology, specifically to a method for dynamic assessment of callus growth during fracture healing based on multimodal data. The method first combines medical assessment indicators of callus growth during fracture healing with the doctor's subjective evaluation of the patient to obtain callus evaluation indicators. It then analyzes the dynamic changes of each modality of data and the callus evaluation indicators to determine the value and relative importance of each modality in callus growth assessment. Based on the value and relative importance of each modality in callus growth assessment, it determines the contribution weight of each modality. Finally, it performs weighted feature fusion of the contribution weight of each modality at each time point with the corresponding feature vector, inputs the resulting fused feature sequence into a time-series model, and outputs the callus growth assessment result. This invention adaptively adjusts the modality weights at different healing stages, improving the assessment quality of callus growth during fracture healing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for dynamic evaluation of callus growth during fracture healing based on multimodal data. Background Technology

[0002] Fractures are a common type of orthopedic injury in clinical practice, with a complex healing process involving multiple stages and various biomechanical and tissue remodeling mechanisms. Callus formation is one of the most crucial steps in fracture healing, the process by which newly formed bony tissue connects the fractured bone ends. With the development of medical imaging, sensing technology, and artificial intelligence, multimodal data integrating X-rays, CT, MRI, ultrasound, biochemical markers, and mechanical signals can comprehensively reflect the repair status of bone tissue from different dimensions, such as structure, function, biochemistry, and mechanics. This allows for dynamic, quantitative, and early assessment of callus formation, further enabling precise monitoring and prediction of the entire fracture healing process.

[0003] In existing technologies, deep networks are typically used to simultaneously learn multiple modalities to assess the callus growth status during fracture healing. These modalities include CT scans, ultrasound, and biochemical indicators. However, during network training, if one modality can quickly assess callus growth (e.g., high-contrast CT images are easier for the network to capture stable features), while other modalities are more difficult to learn and converge more slowly, the network will quickly acquire features from the dominant modality in the early stages. After parameter updates and fixation, it will continue to optimize along this path, leading to a decrease in the model's perceptual channel weights for other modalities. At this point, even if secondary modal data is forcibly input, the model will hardly respond to its features, further trapping itself in a local optimum centered on the dominant modality. Summary of the Invention

[0004] To address the technical problem of decreased weighting of perception channels for other modalities in the model, the present invention aims to provide a method for dynamic evaluation of callus growth during fracture healing based on multimodal data. The specific technical solution adopted is as follows:

[0005] In a first aspect, embodiments of the present invention provide a method for dynamic evaluation of callus growth during fracture healing based on multimodal data, the method comprising:

[0006] Collect fracture-related multimodal data; combine medical assessment indicators of callus growth during the fracture healing period with doctors' subjective evaluations of patients to obtain callus evaluation indicators;

[0007] Analyze the dynamic changes of each modality of data and callus evaluation indicators to determine the value of each modality of data for callus growth assessment;

[0008] The relative importance of each modality of data for the assessment of callus growth is determined based on the dynamic trends of each modality of data and callus evaluation indicators at the same time, the value of each modality of data for the assessment of callus growth, and the differences in the dynamic trends among different modalities of data.

[0009] Based on the value of each modality of data in the assessment of callus growth and the relative importance of each modality of data in the assessment of callus growth, the contribution weight of each modality of data in the assessment of callus growth is determined. The contribution weight of each modality of data at each time point is weighted and fused with the corresponding feature vector to obtain a fused feature sequence. The fused feature sequence is input into a Transformer-based time series model to output the callus growth assessment result.

[0010] Secondly, a dynamic assessment system for callus growth during fracture healing based on multimodal data is provided, the system comprising the following modules:

[0011] The initial evaluation module is used to collect fracture-related multimodal data; combined with medical assessment indicators of callus growth during the fracture healing period and doctors' subjective evaluations of patients, callus evaluation indicators are obtained.

[0012] The dynamic analysis module is used to analyze the dynamic changes of each modality of data and callus evaluation indicators, and to determine the value of each modality of data for callus growth assessment.

[0013] The trend analysis module is used to determine the relative importance of each modality of data for the assessment of callus growth based on the dynamic trends of each modality of data and callus evaluation indicators at the same time, the value of each modality of data for the assessment of callus growth, and the differences in the dynamic trends between different modalities of data.

[0014] The modality weight determination module is used to determine the contribution weight of each modality data to the callus growth assessment based on the value of each modality data to the callus growth assessment and the relative importance of each modality data to the callus growth assessment; the contribution weight of each modality data at each time point is weighted and fused with the corresponding feature vector to obtain a fused feature sequence; the fused feature sequence is input into the Transformer-based time series model to output the callus growth assessment result.

[0015] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the various possible implementations of the first aspect.

[0016] Fourthly, embodiments of the present invention provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0017] Fifthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the various possible implementations of the first aspect.

[0018] The embodiments of the present invention have at least the following beneficial effects:

[0019] In assessing callus growth during fracture healing based on multimodal data, the varying sensitivity of different modalities to the model output (callus growth assessment) and their dependent data distribution and network structure can lead to strong gradient modalities dominating training and suppressing the learning of weak gradient modalities, further reducing the quality of dynamic fracture healing assessment. Therefore, this invention dynamically analyzes the contribution of each modality to the callus evaluation index reflecting fracture healing and adaptively adjusts the contribution weights of each modality at different stages. This fully leverages the advantages of each modality's data at different healing stages, avoids model dependence on a single modality, improves the accuracy, stage sensitivity, and robustness of callus growth assessment, and enhances the early identification of abnormal healing conditions, enabling more detailed and reliable tracking and prediction of the dynamic changes in callus growth.

[0020] By judging the synchronicity between changes in modal data and callus evaluation indicators reflecting callus growth, key modalities that best reflect callus growth status at different healing stages can be selected. This helps the model to focus on data sources with large amounts of information and high sensitivity to change, improving the relevance and accuracy of the overall assessment. Dynamically observing the correlation between modalities and callus growth indicators at each time point reveals the changing roles of each modality in the healing process, avoiding misjudgments caused by a single modality dominating for a long time, and improving the model's ability to model the continuity and sensitivity of callus growth. Systematically comparing the contributions of different modalities in the overall assessment allows for the reasonable allocation of weights for each modality in the fusion process, preventing high signal-to-noise ratio or high-contrast modalities from dominating training, promoting multimodal collaborative learning, and improving the comprehensiveness and robustness of the assessment results. Dynamically calculating the contribution weights of each modality over time not only enables precise weight allocation that evolves over time but also intuitively reflects changes in the healing mechanism at different stages, enhancing the model's deep understanding of the fracture healing process and improving the dynamic assessment quality of callus growth during the fracture healing period.

[0021] Therefore, this invention analyzes the contribution of each modality to fracture healing indicators over time series, adaptively adjusts the weights of each modality at different healing stages, and ensures that the model makes reasonable use of all modal information sources, thereby improving the assessment quality of callus growth during the fracture healing period. Attached Figure Description

[0022] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a method for evaluating the dynamics of callus growth during fracture healing based on multimodal data, provided in one embodiment of the present invention.

[0024] Figure 2 This is a flowchart illustrating a method for assessing the relative importance of each modality to callus growth, as provided in one embodiment of the present invention. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving the intended purpose, the following detailed description, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the method for dynamic evaluation of callus growth during fracture healing based on multimodal data proposed in this invention.

[0026] In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0027] In the description of the embodiments of the present invention, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" means two or more.

[0028] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] The embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0031] The following description, in conjunction with the accompanying drawings, details the specific scheme of the method for dynamic evaluation of callus growth during fracture healing based on multimodal data provided by this invention.

[0032] Please see Figure 1 The diagram illustrates a flowchart of a method for dynamically assessing callus growth during fracture healing based on multimodal data, according to an embodiment of the present invention. The method includes the following steps:

[0033] Step S100: Collect fracture-related multimodal data; combine medical assessment indicators of callus growth during the fracture healing period with doctors' subjective evaluations of patients to obtain callus evaluation indicators.

[0034] Determine the types and time points of the modal data to be collected. Modal data types include CT, MRI images, ultrasound images, and biochemical markers. Time points include postoperative weeks 0, 2, 4, 6, 8, and 12. Develop an electronic CRF template that includes fields for demographics, clinical scores, complications, imaging reports, and laboratory results. Establish necessary editing, verification, and query mechanisms to ensure data consistency and integrity.

[0035] Multimodal data acquisition:

[0036] (1) Imaging:

[0037] X-ray / CT scan: Obtain biplane X-rays and 3D CT scans using fixed parameters (e.g., 120kV, 200mAs; slice thickness 1mm);

[0038] MRI imaging: T2-weighted and contrast-enhanced sequences were used to acquire MRI images;

[0039] Ultrasound examination: B-mode and Doppler blood flow imaging were performed using a 7–12 MHz probe to record the hypoechoic area of ​​the callus and blood perfusion during the inflammatory period, and to acquire ultrasound images.

[0040] (2) Biochemical markers:

[0041] Serum sampling: Collect 5 mL–10 mL of venous blood at each imaging follow-up visit, separate the serum and store at –80°C;

[0042] Indicator detection: Quantitative analysis of osteogenic / osteoclast markers such as PINP, β-CTX, and bone alkaline phosphatase is performed using ELISA or chemiluminescence immunoassay.

[0043] (3) Mechanical properties:

[0044] In vitro mechanical testing: Take specimens for three-point bending and torque-to-failure tests, and record stiffness and ultimate load.

[0045] Furthermore, through built-in edit checks and on-site monitoring, outliers or missing entries in the electronic CRF template can be identified and resolved promptly. Patient information is anonymized to ensure data security and privacy compliance.

[0046] Bone callus growth reflects the repair and remodeling of bone tissue during fracture healing. Different medical modalities provide information at different levels, each with its own unique advantages and limitations. Combining multimodal data can offer a more comprehensive perspective. Therefore, neural networks are often used to train multimodal data to obtain more accurate and reliable bone callus growth assessment results. Because each modality of data describes bone callus growth during fracture healing with different focuses and amounts of information, analyzing the relationship between each modality and bone callus growth can fully explore the unique advantages of each modality, thereby leading to a more accurate assessment of fracture healing.

[0047] Analyze the relationship between data from each modality and callus growth:

[0048] Currently, several medical assessment indicators can be obtained for the callus growth status during the fracture healing period, such as callus volume growth rate (BV / TV), mineralization rate (BMD), and predicted stiffness value, as well as subjective evaluation indicators of patients at each stage by doctors (such as callus growth score). By combining objective and subjective evaluations from several collected fracture healing data, a callus evaluation index can be obtained.

[0049] The assessment of callus growth during the fracture healing period in the case training set is based on multimodal data to obtain objective medical assessment indicators, such as callus volume increase / TV, mineralization rate (BMD), and predicted stiffness value, all of which are obtained through multimodal synthesis. In this embodiment of the invention, the callus volume increase is assessed by multiple imaging data, the mineralization rate is assessed by ultrasound, CT, and X-ray, and the predicted stiffness value, which is the mechanical stiffness of the callus or fracture area, is obtained by multiple imaging and mechanical experiments. However, due to the limitations of imaging in different case data, the accuracy of objective indicators may have errors. The above objective and subjective evaluation results need to be judged and evaluated by senior attending physicians to derive callus evaluation indicators, which serve as a reference for the training dataset. During the training process, subsequent multimodal data are processed and weighted to make the callus growth assessment results more accurate and better match the assessment conclusions of high-level physicians.

[0050] When senior chief physicians clinically evaluate callus growth, their assessments at different time series are more informed by experience regarding which modal data to emphasize. In contrast, general models are often influenced by dominant modal data during training, leading to limitations in the assessment. By comparing physician evaluations with reference data, the dynamic contribution weights obtained will better align with the physician's assessment, eliminating dominant dependence and yielding more accurate results.

[0051] Therefore, the subjective evaluation and medical assessment indicators in the fracture healing medical data under the callus growth status during the fracture healing period are normalized in order to standardize the values ​​of all subjective evaluations and medical assessment indicators.

[0052] The mean of the subjective evaluation and the normalized results of the medical assessment indicators at each time point are used as the callus evaluation indicator for each time point.

[0053] In some embodiments, the callus evaluation index for each time point is calculated to obtain the callus evaluation index for each time point: Where Y represents the callus evaluation index, reflecting the callus growth during the fracture healing period; A represents the total number of medical assessment indicators and subjective evaluations recorded in the medical records for evaluating callus growth; D a Let represent the a-th evaluation indicator; sigmoid is the normalization function. The evaluation indicators include: medical evaluation indicators and subjective evaluations.

[0054] Step S200: Analyze the dynamic changes of each modal data and callus evaluation index to determine the value of each modal data for callus growth assessment.

[0055] In the process of dynamically evaluating callus growth using multimodal data trained through neural networks, the difficulty in setting the weights of each modality leads to the problem of certain modalities dominating the training. Callus growth is a dynamic process, and the characteristics of callus and related modal data may change at different stages of healing. By analyzing the relationship between different modalities and callus growth, the weights of each modality can be dynamically adjusted to ensure that the data of each modality is used appropriately at different stages.

[0056] A convolutional neural network (CNN) is used to perform independent feature extraction for each modality of data. The feature extractor uses the time-series modality data to form time-series features corresponding to each modality, fully recording the dynamic changes during the callus growth process. For example, it extracts the density distribution features within the callus region of CT images and their patterns of change over time. Each modality of data will correspond to multiple feature value sequences because there may be multiple features reflecting callus growth.

[0057] By analyzing the dynamic changes of each modality of data and callus evaluation indicators, the value of each modality of data for callus growth assessment is determined. More specifically: after obtaining the feature value sequence corresponding to each modality of data, any type of modality of data is used as the target modality of data; any feature value sequence of the target modality of data is used as the target feature value sequence; the dynamic matching distance between the target feature value sequence and the callus evaluation indicator sequence is calculated; a negative correlation mapping is performed on the dynamic matching distance to obtain the dynamic matching value; the mean of the dynamic matching values ​​corresponding to all feature value sequences of the target modality of data is used as the value of the target modality of data for callus growth assessment.

[0058] In some embodiments, the i-th modal data is used as the target modal data, and the value R of the target modal data for assessing callus growth is... i The calculation formula is: Among them, M i T represents the number of feature value sequences extracted from the i-th modality data through the neural network; i,j Let represent the sequence of the j-th feature value in the i-th modality of data; Y represents the sequence of callus evaluation indicators. DTW(T i,j (T, Y) represents the dynamic matching distance between the j-th feature value sequence and the callus evaluation index sequence in the i-th modality data. It reflects the consistency of the changes between the j-th feature value sequence and the callus evaluation index sequence over time. This dynamic matching distance is the DTW matching distance, DTW(T i,j The smaller the value of Y, the more consistent the modal data is with changes in callus growth.

[0059] Step S300: Based on the dynamic trends of each modal data and callus evaluation indicators at the same time, the value of each modal data for callus growth assessment, and the differences in dynamic trends among different modal data, determine the relative importance of each modal data for callus growth assessment.

[0060] It is known that callus formation is a dynamic process, with different bone repair mechanisms and physiological characteristics at different stages. The early stage is dominated by inflammation and blood supply remodeling, the middle stage by new bone formation, and the late stage by bone remodeling. Different modalities reflect different biological information at different stages, and the advantage of modal characteristics in assessing callus formation changes over time. Dynamic analysis of correlation trends is needed to capture the importance of each modality at different stages of fracture healing. Furthermore, different patients have different fracture types, repair speeds, and complication rates, resulting in varying values ​​for information from different modalities. Dynamic correlation trend analysis allows the model to adjust its decision-making strategies based on actual data, improving its adaptability and assessment accuracy across different patients and fracture types.

[0061] For example, CT images with dominant modal signals are of high value for assessing callus growth. However, in the early cartilage callus stage of fracture healing, ultrasound signals are more sensitive to assessing callus growth, and it is necessary to dynamically analyze the importance of each modality for assessing callus growth.

[0062] The dynamic correlation between modal characteristics and callus growth is analyzed to obtain the relative importance of each modality for callus growth assessment. In this embodiment of the invention, callus growth is characterized by callus evaluation indicators. Therefore, the relative importance of each modality for callus growth assessment is determined by the dynamic trends of each modality's data and callus evaluation indicators at the same time, the value of each modality's data for callus growth assessment, and the differences in the dynamic trends between different modalities.

[0063] Please see Figure 2 , Figure 2 A flowchart illustrating the method for assessing the relative importance of each modality to callus growth; the method for obtaining the relative importance of each modality to callus growth assessment includes the following steps:

[0064] Step S310: Based on the dynamic trends of each modal data and callus evaluation index at the same time, and combined with the value of each modal data for callus growth assessment, determine the consistency of the dynamic trends of each modal data and callus evaluation index.

[0065] Bone callus growth changes, and so does its modal value. Only by dynamically analyzing the correlation trends can we truly keep up with the pace of fracture healing and make accurate and reliable dynamic assessments of callus growth. This involves analyzing the consistency of the dynamic trends of modal characteristics and comprehensive evaluation indicators at the same time point.

[0066] Using any type of modal data as the target modal data, the trend value of the target modal data is determined based on its changing trend over the analysis period. The trend value of the callus evaluation index is also determined based on its changing trend over the analysis period. The difference between the trend values ​​of the target modal data and the callus evaluation index is negatively correlated. The product of the negative correlation mapping result and the value of the target modal data for assessing callus growth is then used to determine the consistency of the dynamic trends of the modal data and the callus evaluation index.

[0067] In some embodiments, within the neighborhood of each modality data point corresponding to the time to be analyzed, the slope of the fitted line corresponding to each modality data point is obtained using the least squares principle as the trend value of the change of each modality data point at the time to be analyzed. In this embodiment of the invention, the neighborhood of the time to be analyzed includes the three time points before the time to be analyzed and the time to be analyzed. It should be noted that the method for obtaining the trend value of the callus evaluation index is the same: within the neighborhood of the time to be analyzed, the slope of the fitted line corresponding to the callus evaluation index is obtained using the least squares principle as the trend value of the callus evaluation index at the time to be analyzed.

[0068] In some embodiments, the i-th modal data is used as the target modal data, and the o-th time point is used as the time to be analyzed. The formula for calculating the consistency of the dynamic trend of the i-th modal data with the callus growth evaluation index at the o-th time point is: Q i,o =R i,o ×exp(-norm(|X i,o -X o |));where Q i,o This indicates the consistency of the dynamic trend of the i-th modality data with the comprehensive evaluation index of callus growth at time point o; R i,o X represents the value of the i-th modality data for assessing callus growth at time point o; i,o X represents the trend value of the i-th modality data at time point o; o The value representing the trend of callus evaluation index at time point 0 is |X. i,o -X o | represents the difference between the trend value of the i-th modal data and the callus evaluation index at the o-th time point, when |X i,o -X o The smaller the value, the more synchronized the changing trends, and the greater the consistency of their dynamic trends; exp(-norm(|X i,o -X o |)) is to achieve |X i,o -X o The negative correlation mapping of | is also the mapping of |X.i,o -X o | The result of performing negative correlation mapping.

[0069] Step S320: Determine the dynamic correlation trend between each modal data and the callus evaluation index based on the consistency and partial correlation coefficient of the dynamic trend between each modal data and the callus evaluation index.

[0070] When the evaluation indicators for callus growth are influenced by multimodal data, it is necessary to control for other modal characteristics, extract confounding effects, and obtain only the net correlation between the target modality and the evaluation indicators. By combining the net correlation with the consistency of dynamic trends, the dynamic correlation trend between each modality and the evaluation indicators can be obtained.

[0071] Specifically: For the target modal data, calculate the partial correlation coefficient between the target modal data and the callus evaluation index; take the product of the dynamic trend consistency and the partial correlation coefficient corresponding to the target modal data as the dynamic correlation trend between the target modal data and the callus evaluation index.

[0072] The formula for calculating the dynamic correlation trend of the i-th modality data at time point o is: G i,o =Q i,o ×P i,o Q i,o This indicates the consistency of the dynamic trend of the i-th modality data with the callus evaluation index at time point o; P i,o This represents the partial correlation coefficient between the data of the i-th modality at the o-th time point and the callus evaluation index for callus growth.

[0073] Step S330: Combining the value of each modality data for callus growth assessment and the differences in the dynamic correlation trends between different modal data, determine the relative importance of each modality for callus growth assessment.

[0074] In the assessment of fracture healing, different modalities capture information about a certain aspect of callus growth. By comparing the predictive or explanatory abilities of these modalities for callus growth indicators at the same time point, the independent contributions and complementary values ​​of each modality can be revealed, thereby obtaining the relative importance of each modality for growth assessment.

[0075] By comparing the dynamic correlation trends of different modalities at the same time point, when a single modality shows an increasing dynamic correlation trend while other modalities show moderate or low dynamic correlation trends, that modality is relatively important for growth assessment at that moment. Any type of modal data is used as the target modal data; at the time to be analyzed, submodal reference values ​​are determined based on the differences in the dynamic correlation trends between the target modal data and other modal data. Using the i-th modal data as the target modal data, at the 0-th time point, the submodal reference value ΔG corresponding to the target modal data and the n-th modal data is... i,n,oThe calculation formula is: ΔG i,n,o =norm(G i,o -G n,o );G i,o This represents the dynamic correlation trend between the data of the i-th modality at time point o and the callus evaluation index of callus growth; G n,o This represents the dynamic correlation trend between the data of the nth modality at the 0th time point and the callus evaluation index of callus growth.

[0076] The differences between each modality at the overall and different fracture healing stages can reflect the necessity of adjusting the weight of that modality at this moment. If the value of other modalities to growth indicators differs from that of the whole at the current time point, it indicates that their current weights are inappropriate.

[0077] By comparing the value of other modal data in assessing callus growth during the analysis period with the overall time period, the modal change weights of other modal data during the analysis period are determined. Specifically: the mean value of other modal data in assessing callus growth during the overall time period is calculated as the overall assessment value; the normalized value of the difference between the value of other modal data in assessing callus growth during the analysis period and the overall assessment value is used as the numerator, and the overall assessment value is used as the denominator. The ratio of the numerator and denominator is used as the modal change weight of other modal data during the analysis period.

[0078] In some embodiments, taking the nth modal data as an example, the modal change weight C of the nth modal data at time point o is... n,o The calculation formula is: Among them, R n,o R represents the value of the nth modality data for assessing callus growth at time point o; n This represents the overall evaluation value of the nth modality data; norm is the normalization function.

[0079] If a modality shows a stronger dynamic correlation with callus growth evaluation indicators compared to other modalities at the same time point, that modality is more important for growth assessment. Furthermore, when other modalities show a need for weight adjustment, the greater the change in weight of the other modalities, the greater their relative importance compared to other modalities. The relative importance of each modality for callus growth assessment is obtained by comparing data from other modalities.

[0080] Combining the submodal reference values ​​and modal change weights, the relative importance of each modal data to callus growth assessment is determined. Specifically: the product of the submodal reference values ​​and modal change weights between the target modal data and other modal data is calculated as the submodal importance of the target modal data to other modal data; the sum of the submodal importance of the target modal data and all modal data is used as the numerator, and the sum of the submodal reference values ​​between the target modal data and other modal data is used as the denominator. The ratio of the numerator and denominator is used as the relative importance of the target modal data to callus growth assessment.

[0081] In some embodiments, the i-th modal data is still used as the target modal data, and the relative importance of the target modal data for callus growth assessment at the o-th time point is B. i,o The calculation formula is: Among them, C n,o ΔG represents the weight of the change in the nth modality at time point 0; i,n,o C represents the submodal reference value of the i-th modal data and the n-th modal data at time point 0; n,o ×ΔG i,n,o Let represent the submodal importance of the i-th modal data and the n-th modal data at time point 0; N is the number of modal data types in the multimodal data.

[0082] Step S400: Based on the value of each modality of data in the assessment of callus growth and the relative importance of each modality of data in the assessment of callus growth, determine the contribution weight of each modality of data in the assessment of callus growth; perform weighted feature fusion with the corresponding feature vector at each time point to obtain a fused feature sequence; input the fused feature sequence into the Transformer-based time series model to output the callus growth assessment result.

[0083] Multimodal fusion is prone to being dominated by a dominant modality. If the weights of each modality are not dynamically adjusted, the model may over-rely on the modality features that dominate the assessment of callus growth during fracture healing, while ignoring some biologically crucial modal signals in other healing stages. Therefore, it is essential to adjust the contribution ratio of each modality during fusion according to its relative importance to callus growth assessment, avoiding bias caused by a single dominant modality.

[0084] Existing multimodal fusion methods for dynamic assessment of callus growth during fracture healing often involve simple modal feature splicing or assigning fixed or coarsely adjusted static weights to each modality based on modal feature matching and complementarity. These methods neglect the dynamic nature of fracture healing, or calculate time-series dynamic weights for independent features, ignoring the correlations between modal features. This fails to address the issue of the dominant modality signal influencing the assessment quality of other healing stages in callus growth dynamic assessment. Therefore, this paper derives the contribution weight of each modality to callus growth assessment by considering the value of each modality's data to callus growth assessment throughout the overall fracture healing period and the relative importance of each modality at each healing time point. Specifically, the product of the value of each modality's data to callus growth assessment and its relative importance is used as the contribution weight of each modality's data to callus growth assessment.

[0085] In some embodiments, the contribution weight Z of the i-th modality data to the callus growth assessment at time point o is... i,o The calculation formula for Z is: i,o =R i ×B i,o Among them, R i B represents the value of the i-th modality data for assessing callus growth, and is the comprehensive value throughout the entire fracture healing period; i,o This represents the relative importance of the i-th modality of data at time point o for the assessment of callus growth. The contribution weight indicates that, based on the overall value of assessing callus growth, the higher the relative importance of the i-th modality of data at time point o, the more dynamically its contribution weight at that time point is increased.

[0086] The contribution weights of each modality at each time point are weighted and aggregated with their corresponding feature vectors to dynamically highlight information from key modalities. It should be noted that the feature vectors corresponding to the multimodal data are obtained by inputting the multimodal data into a convolutional neural network, resulting in feature vectors of uniform dimension, where each feature value corresponds to one modality of data.

[0087] After obtaining the adaptive contribution weights of each modality at each time point and completing the weighted feature fusion, the fused feature sequence at several consecutive time points is input into the Transformer-based time series model to capture the long-term and short-term dependencies and dynamic patterns of the callus growth process. The time series model outputs the callus growth assessment results and compares the generated time series prediction values ​​with historical observations to achieve a quantitative assessment of the fracture healing process. The time series modeling capability is further optimized through loss function feedback.

[0088] New multimodal data is fed back to update the weight learning strategy online, forming a closed-loop adaptive evaluation system that is continuously optimized as the data is updated.

[0089] Input new fracture multimodal data for callus growth assessment:

[0090] (1) The clinical or experimental platform imports multimodal data of patients’ CT, MRI, ultrasound images and serum biochemical test results at the latest follow-up visit into the system. Each modality of data is accurately timestamped and mapped to a predetermined unified time point based on a Gaussian process.

[0091] (2) Input multimodal data into a convolutional neural network to obtain a feature vector of uniform dimension.

[0092] (3) According to the above steps S100 to S400, the current features of each modality data are spliced ​​together, the contribution weight of each modality at this time point is output, the weighted fusion modality features are input into the Transformer temporal network, and the comprehensive evaluation results of callus growth at the next time point and throughout the entire follow-up period are output.

[0093] This invention provides a dynamic assessment system for callus growth during fracture healing based on multimodal data. The system includes:

[0094] The initial evaluation module is used to collect fracture-related multimodal data; combined with medical assessment indicators of callus growth during the fracture healing period and doctors' subjective evaluations of patients, callus evaluation indicators are obtained.

[0095] The dynamic analysis module is used to analyze the dynamic changes of each modality of data and callus evaluation indicators, and to determine the value of each modality of data for callus growth assessment.

[0096] The trend analysis module is used to determine the relative importance of each modality of data for the assessment of callus growth based on the dynamic trends of each modality of data and callus evaluation indicators at the same time, the value of each modality of data for the assessment of callus growth, and the differences in the dynamic trends between different modalities of data.

[0097] The modality weight determination module is used to determine the contribution weight of each modality data to the callus growth assessment based on the value of each modality data to the callus growth assessment and the relative importance of each modality data to the callus growth assessment; the contribution weight of each modality data at each time point is weighted and fused with the corresponding feature vector to obtain a fused feature sequence; the fused feature sequence is input into the Transformer-based time series model to output the callus growth assessment result.

[0098] Optionally, the transmission medium can be a wired link, such as, but not limited to, coaxial cable, fiber optic cable and digital subscriber line, or a wireless link, such as, but not limited to, wireless Fidelity (WIFI), Bluetooth and mobile device networks.

[0099] It should be noted that the device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above.

[0100] Furthermore, embodiments of the present invention also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform the dynamic evaluation method for callus growth during fracture healing based on multimodal data provided in embodiments of the present invention.

[0101] In this embodiment of the invention, the device can be divided into functional modules according to the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0102] When each module is divided according to its function, the device may also include a signal uploading module, a determination module, and an adjustment module. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0103] It should be understood that the device provided in this embodiment of the invention is used to perform the above-described method for dynamic evaluation of callus growth during fracture healing based on multimodal data, and therefore can achieve the same effect as the above-described implementation method.

[0104] When using integrated units, the device may include a processing module and a storage module. When applied to a device, the processing module can be used to control and manage the device's operations. The storage module can be used to support the device in executing program code, etc. The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as described in this disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of Digital Signal Processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0105] In addition, the device provided in the embodiments of the present invention may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute the dynamic evaluation method for callus growth during fracture healing based on multimodal data provided in the above embodiments.

[0106] This invention also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement the fracture healing period callus growth dynamic assessment method based on multimodal data provided in the above embodiments.

[0107] This invention also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the method for dynamic evaluation of callus growth during fracture healing based on multimodal data provided in the above embodiments.

[0108] In this invention, the apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments are all used to execute the corresponding methods described above. Therefore, the beneficial effects they achieve can be referred to the beneficial effects in the corresponding methods described above, and will not be repeated here. Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways.

[0109] The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0110] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0111] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0113] The above content is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamically assessing callus growth during fracture healing based on multimodal data, characterized in that, The method includes the following steps: Collect fracture-related multimodal data; combine medical assessment indicators of callus growth during the fracture healing period with doctors' subjective evaluations of patients to obtain callus evaluation indicators; Analyze the dynamic changes of each modality of data and callus evaluation indicators to determine the value of each modality of data for callus growth assessment; Based on the dynamic trends of each modality of data and callus evaluation indicators at the same time, the value of each modality of data for callus growth assessment, and the differences in dynamic trends among different modalities of data, the relative importance of each modality of data for callus growth assessment is determined. This includes: determining the consistency of the dynamic trends of each modality of data and callus evaluation indicators based on the dynamic trends of each modality of data and callus evaluation indicators at the same time, combined with the value of each modality of data for callus growth assessment; determining the dynamic correlation trend of each modality of data and callus evaluation indicators based on the consistency of the dynamic trends of each modality of data and callus evaluation indicators and the partial correlation coefficient; and determining the relative importance of each modality of data for callus growth assessment by combining the value of each modality of data for callus growth assessment and the differences in dynamic correlation trends among different modalities of data. Based on the value of each modality of data in the assessment of callus growth and the relative importance of each modality of data in the assessment of callus growth, the contribution weight of each modality of data in the assessment of callus growth is determined. The contribution weight of each modality of data at each time point is weighted and fused with the corresponding feature vector to obtain a fused feature sequence. The fused feature sequence is input into a Transformer-based time series model to output the callus growth assessment result.

2. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The medical assessment indicators for callus growth during the fracture healing period, combined with the doctor's subjective evaluation of the patient, are used to obtain callus evaluation indicators, including: Normalize the subjective evaluation and medical assessment indicators in fracture healing medical data on callus growth during the fracture healing period. The mean of the subjective evaluation and the normalized results of the medical assessment indicators at each time point are used as the callus evaluation indicator for each time point.

3. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The analysis of the dynamic changes of each modality of data and callus evaluation indicators, and the determination of the value of each modality of data for callus growth assessment, includes: The Convolutional Neural Network (CNN) is used to perform independent feature extraction for each modality of data. The feature extractor forms time-series features for each modality of data through the time-series modality data, and each modality of data will correspond to multiple feature value sequences. Using any type of modal data as target modal data; using any sequence of feature values ​​of the target modal data as target feature value sequence; calculating the dynamic matching distance between the target feature value sequence and the callus evaluation index sequence; performing negative correlation mapping on the dynamic matching distance to obtain the dynamic matching value; The mean of the dynamic matching values ​​corresponding to all feature value sequences of the target modality data is used as the value of the target modality data for evaluating callus growth.

4. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The determination of the consistency of the dynamic trends of each modality of data and the callus evaluation index at the same time, combined with the value of each modality of data for callus growth assessment, includes: Using any type of modal data as the target modal data, the trend value of the target modal data is determined based on the trend of the target modal data over the time to be analyzed. Based on the changing trend of the callus evaluation index over the analysis period, determine the changing trend value of the callus evaluation index; The difference between the trend values ​​of the target modal data and the callus evaluation index is negatively correlated. The product of the negative correlation mapping result and the value of the target modal data for callus growth assessment is taken as the dynamic trend consistency between the modal data and the callus evaluation index.

5. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The determination of the dynamic correlation trend between each modality of data and the callus evaluation index based on the consistency and partial correlation coefficient of the dynamic trend between each modality of data and the callus evaluation index includes: Using any type of modal data as the target modal data, calculate the partial correlation coefficient between the target modal data and the callus evaluation index; The product of the dynamic trend consistency and partial correlation coefficient corresponding to the target modal data is used as the dynamic correlation trend between the target modal data and the callus evaluation index.

6. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The determination of the relative importance of each modality for callus growth assessment, by combining the value of each modality of data for callus growth assessment and the differences in the dynamic correlation trends between different modalities, includes: Any type of modal data can be used as the target modal data; at the time to be analyzed, the submodal reference values ​​are determined based on the differences in the dynamic correlation trends between the target modal data and other types of modal data. Compare the value of other modal data for assessing callus growth during the time period to be analyzed and the overall time period to determine the modal change weights of other modal data during the time period to be analyzed; By combining the submodal reference values ​​and modal change weights, the relative importance of each modal data for callus growth assessment is determined.

7. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 6, characterized in that, The comparison of the value of other modal data for assessing callus growth within the time period to be analyzed and the overall time period, and the determination of the modal change weights of other modal data within the time period to be analyzed, includes: The mean value of other modal data for assessing callus growth over the entire time period is calculated as the overall assessment value; The normalized value of the difference between the value of other modal data for callus growth assessment at the time to be analyzed and the overall assessment value is used as the numerator, and the overall assessment value is used as the denominator. The ratio of the numerator and denominator is used as the modal change weight of other modal data at the time to be analyzed.

8. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 6, characterized in that, The determination of the relative importance of each modality data for callus growth assessment, by combining the submodal reference values ​​and modal change weights, includes: The product of the submodal reference value and the modal change weight between the target modal data and other modal data is calculated as the submodal importance of the target modal data and other modal data. The sum of the submodal importance of the target modal data and all modal data is used as the numerator, and the sum of the submodal reference values ​​between the target modal data and other modal data is used as the denominator. The ratio of the numerator and denominator is used as the relative importance of the target modal data for callus growth assessment.

9. The method for dynamic evaluation of callus growth during fracture healing based on multimodal data according to claim 1, characterized in that, The determination of the contribution weight of each modality of data to the assessment of callus growth, based on the value of each modality of data in the assessment of callus growth and the relative importance of each modality of data in the assessment of callus growth, includes: The product of the value of each modality in the assessment of callus growth and the relative importance of each modality in the assessment of callus growth is used as the contribution weight of each modality in the assessment of callus growth.

Images