Bone injury nursing effect evaluation method and system based on multi-modal data fusion

Abstract

The application provides a bone injury nursing effect evaluation method and system based on multi-modal data fusion, pre-processes multi-modal data in bone injury nursing to obtain a multi-modal time series tensor; extracts deep features of image data to obtain an imaging feature vector, and extracts mechanical features of motion data to obtain a mechanical feature vector; constructs a nursing digital twin based on the imaging feature vector and the mechanical feature vector, performs multi-modal fusion evaluation through a current state of the nursing digital twin to obtain a comprehensive nursing evaluation result; constructs an imaging contribution heat map and a mechanical contribution heat map according to the comprehensive nursing evaluation result, and dynamically updates the nursing digital twin based on the imaging contribution heat map and the mechanical contribution heat map. The technical scheme provided by the application can construct a nursing digital twin with an explanatory output based on contribution visualization and a closed-loop updating mechanism, so as to improve the explainability and accuracy of bone injury nursing effect evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of nursing assessment, more particularly, the present application relates to a bone injury nursing effect assessment method and system based on multi-modal data fusion. BACKGROUND

[0002] The diagnosis and treatment and rehabilitation nursing of bone injury is an important field of intersection of clinical medicine and rehabilitation engineering. With the development of medical image processing, wearable sensing and artificial intelligence technology, the industry gradually proposes an automatic evaluation method based on image analysis or based on kinematic analysis. The nursing process of bone injury patients can produce and record a large amount of multi-modal data, which covers medical images (such as X-ray, CT) reflecting the changes of bone anatomical structure, motion and mechanical data (such as gait, joint range of motion information collected by wearable sensors) reflecting the functional recovery status of patients, etc. In theory, the deep fusion and intelligent analysis of these multi-modal data can realize the accurate, objective and dynamic evaluation of the nursing effect of bone injury.

[0003] In the prior art, simple splicing or weighted average and other rough strategies are often used, which lack the ability to organically couple static structure information and dynamic function information to jointly drive individualized prediction and intervention evaluation, and the prior art lacks a digital twin that can continuously incorporate multi-source time series data and dynamically calibrate and iterate its internal parameters accordingly. Therefore, how to construct a nursing digital twin based on contribution visualization of explanatory output and closed-loop updating mechanism to improve the explainability and accuracy of bone injury nursing effect evaluation is a difficult problem faced by the industry. SUMMARY

[0004] The present application provides a bone injury nursing effect assessment method and system based on multi-modal data fusion, which can construct a nursing digital twin based on contribution visualization of explanatory output and closed-loop updating mechanism to improve the explainability and accuracy of bone injury nursing effect evaluation.

[0005] In a first aspect, the present application provides a bone injury nursing effect assessment method based on multi-modal data fusion, comprising the following steps:

[0006] Obtaining multi-modal data in bone injury nursing, pre-processing the multi-modal data to obtain a multi-modal time series tensor;

[0007] Performing deep feature extraction on image data in the multi-modal time series tensor to obtain an imaging feature vector, and performing mechanical feature extraction on motion data in the multi-modal time series tensor to obtain a mechanical feature vector;

[0008] construct a nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector, perform multi-modal fusion evaluation through a current state of the nursing digital twin, and obtain a comprehensive nursing evaluation result;

[0009] construct an imaging contribution heat map and a mechanical contribution heat map according to the comprehensive nursing evaluation result, and dynamically update the nursing digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map.

[0010] In some embodiments, the multi-modal data includes imaging data and motion data in bone injury care.

[0011] In some embodiments, the pre-processing of the multi-modal data is time stamp alignment and missing value filling of the multi-modal data, thereby obtaining a multi-modal time series tensor.

[0012] In some embodiments, the deep feature extraction of the imaging data in the multi-modal time series tensor includes:

[0013] performing pixel pre-processing on the imaging data in the multi-modal time series tensor, and then performing image deep segmentation on the pixel pre-processed imaging data to obtain a pixel-level segmentation mask;

[0014] extracting fracture features from the pixel-level segmentation mask to obtain callus features, fracture line clarity, and skeletal alignment and alignment geometry features;

[0015] constructing an imaging feature vector based on the callus features, the fracture line clarity, and the skeletal alignment and alignment geometry features.

[0016] In some embodiments, the mechanical feature extraction of the motion data in the multi-modal time series tensor includes:

[0017] calibrating the motion data in the multi-modal time series tensor, and then performing gait cycle detection on the calibrated motion data to obtain a gait cycle sequence;

[0018] determining basic gait parameters of each gait cycle in the gait cycle sequence;

[0019] determining mechanical features and variability indexes of each gait cycle according to the corresponding basic gait parameters;

[0020] constructing a mechanical feature vector through the mechanical features and variability indexes of all gait cycles.

[0021] In some embodiments, constructing a nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector specifically includes:

[0022] time aligning and standardizing the imaging feature vector and the mechanical feature vector to obtain a state vector in bone injury care;

[0023] constructing a parameterized agent model through the state vector;

[0024] individualizing parameter adjustment on the agent model to obtain a nursing digital twin in bone injury care.

[0025] In some embodiments, the multi-modal fusion evaluation through the current state of the nursing digital twin is to input the current state of the nursing digital twin as input data into a multi-modal evaluation model for evaluation, and then obtain a comprehensive nursing evaluation result.

[0026] In a second aspect, the present application provides a bone injury care effect evaluation system based on multi-modal data fusion, which is used to execute a bone injury care effect evaluation method based on multi-modal data fusion, and includes:

[0027] a data acquisition module, configured to acquire multi-modal data in bone injury care, and to obtain a multi-modal time series tensor by preprocessing the multi-modal data;

[0028] a feature extraction module, configured to extract deep features from image data in the multi-modal time series tensor to obtain an imaging feature vector, and to extract mechanical features from motion data in the multi-modal time series tensor to obtain a mechanical feature vector;

[0029] a fusion evaluation module, configured to construct a nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector, and to perform multi-modal fusion evaluation through the current state of the nursing digital twin to obtain a comprehensive nursing evaluation result;

[0030] a dynamic updating module, configured to construct an imaging contribution heat map and a mechanical contribution heat map based on the comprehensive nursing evaluation result, and to dynamically update the nursing digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map.

[0031] In a third aspect, the present application provides a computer device, which includes a memory and a processor, the memory stores a code, and the processor is configured to acquire the code and execute the above-mentioned bone injury care effect evaluation method based on multi-modal data fusion.

[0032] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned bone injury care effect evaluation method based on multi-modal data fusion.

[0033] The technical scheme provided by the embodiments disclosed in the application has the following beneficial effects:

[0034] In the method and system for evaluating the effect of bone injury nursing based on multi-modal data fusion provided by the application, multi-modal data in bone injury nursing is acquired, the multi-modal data is preprocessed to obtain a multi-modal time series tensor, image data in the multi-modal time series tensor is subjected to deep feature extraction to obtain an image feature vector, and motion data in the multi-modal time series tensor is subjected to mechanical feature extraction to obtain a mechanical feature vector; a nursing digital twin in bone injury nursing is constructed based on the image feature vector and the mechanical feature vector, multi-modal fusion evaluation is performed based on the current state of the nursing digital twin, and a comprehensive nursing evaluation result is obtained; an image contribution heat map and a mechanical contribution heat map are respectively constructed according to the comprehensive nursing evaluation result, and the nursing digital twin in bone injury nursing is dynamically updated based on the image contribution heat map and the mechanical contribution heat map.

[0035] As can be seen, in the application, first, the image data and the motion data in the multi-modal time series tensor are subjected to deep feature extraction respectively, which can convert the structural change information of fracture healing and the dynamic performance of patient functional recovery from original high-dimensional data into stable, quantifiable and fusable representations; then, the image feature vector and the mechanical feature vector are used together to construct the nursing digital twin, and multi-modal fusion evaluation is performed based on the current state of the digital twin, which can integrate complementary information in a unified parameterized physiological structure-function model, thereby significantly improving the accuracy of the comprehensive evaluation result; finally, the image contribution heat map and the mechanical contribution heat map are constructed according to the comprehensive nursing evaluation result, and the nursing digital twin is dynamically updated using the two types of heat maps, which can explicitly express the key influencing factors in the comprehensive evaluation result and directly act on the internal parameters of the nursing digital twin, so that the sensitivity and accuracy of the digital twin to structural healing indicators and functional mechanical features are optimized in a targeted manner, and through the updating strategy based on contribution visualization, the nursing digital twin can provide interpretable evaluation output, thereby significantly improving the accuracy and interpretability of the evaluation of the effect of bone injury nursing.

[0036] In summary, the technical scheme adopted in the application can construct a nursing digital twin with an interpretable output based on contribution visualization and a closed-loop updating mechanism to improve the interpretability and accuracy of the evaluation of the effect of bone injury nursing. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required by the embodiments or prior art description. Obviously, the drawings in the following description only represent a part of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort based on these drawings.

[0038] Figure 1 is an exemplary flowchart of a bone injury nursing effect evaluation method based on multi-modal data fusion according to some embodiments of the present application;

[0039] Figure 2 is an exemplary flowchart of determining an imaging feature vector according to some embodiments of the present application;

[0040] Figure 3 is a structural schematic diagram of a bone injury nursing effect evaluation system based on multi-modal data fusion according to some embodiments of the present application;

[0041] Figure 4 is a structural schematic diagram of a computer device for implementing a bone injury nursing effect evaluation method based on multi-modal data fusion according to some embodiments of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent a part of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the scope of protection of the present application.

[0043] The present application provides a bone injury nursing effect evaluation method and system based on multi-modal data fusion. The core is to obtain multi-modal data in bone injury nursing, pre-process the multi-modal data to obtain a multi-modal time series tensor, extract deep features from the imaging data in the multi-modal time series tensor to obtain an imaging feature vector, extract mechanical features from the motion data in the multi-modal time series tensor to obtain a mechanical feature vector, construct a nursing digital twin in bone injury nursing based on the imaging feature vector and the mechanical feature vector, perform multi-modal fusion evaluation through the current state of the nursing digital twin to obtain a comprehensive nursing evaluation result, construct an imaging contribution heat map and a mechanical contribution heat map based on the comprehensive nursing evaluation result, and dynamically update the nursing digital twin in bone injury nursing based on the imaging contribution heat map and the mechanical contribution heat map. The above scheme can construct a nursing digital twin with interpretive output and closed-loop updating mechanism based on contribution visualization, so as to improve the interpretability and accuracy of bone injury nursing effect evaluation.

[0044] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments. Referring to Figure 1 The figure is an exemplary flowchart of a bone injury nursing effect evaluation method based on multi-modal data fusion according to some embodiments of the present application, which mainly includes the following steps:

[0045] In step S101, multi-modal data in bone injury nursing is obtained, and the multi-modal data is preprocessed to obtain a multi-modal time series tensor.

[0046] It should be noted that in the present application, the multi-modal data includes image data and motion data in bone injury nursing. In specific implementation, first, the injured part is periodically imaged and collected by a bedside imaging device, a portable ultrasound instrument or a follow-up film shooting system to obtain image sequences such as fracture alignment, soft tissue swelling and callus formation, i.e. image data in bone injury nursing; then, the motion data such as limb segment motion trajectory, joint angle change and dynamic force during training is synchronously collected by using wearable inertial sensors, surface electromyography patches or rehabilitation monitoring cameras, so as to obtain motion data in bone injury nursing.

[0047] In some embodiments, the pre-processing of the multi-modal data is time stamp alignment and missing value filling of the multi-modal data, and then a multi-modal time series tensor is obtained.

[0048] In specific implementation, first, a unified nursing period time axis can be constructed according to the original time stamps recorded by the image collection device and the motion sensor, and each modal data is mapped to the unified time axis to realize accurate alignment of the image modal and the motion modal in the time dimension. Then, for the time sequence gaps on the time axis caused by image interval collection or sensor instantaneous drop, interpolation inference, time series smoothing or trend compensation based on neighborhood dynamics characteristics can be used to fill the missing data points, so that the multi-modal sequence remains unified in continuity and dimension consistency. Finally, the image feature sequence and the motion feature sequence after time alignment and missing completion can be stacked or spliced according to the time dimension to construct a multi-modal time series tensor containing multi-source information and having complete time series structure, which provides a standardized input for the subsequent nursing effect evaluation model.

[0049] In step S102, deep feature extraction is performed on the image data in the multi-modal time series tensor to obtain an imaging feature vector, and mechanical feature extraction is performed on the motion data in the multi-modal time series tensor to obtain a mechanical feature vector.

[0050] In some embodiments, referring to Figure 2As shown, the figure is an exemplary flowchart for determining an imaging feature vector according to some embodiments of the present application. In this embodiment, the following steps can be used to extract deep features from the image data in the multi-modal time-series tensor to obtain an imaging feature vector:

[0051] In step S1021, the image data in the multi-modal time-series tensor is pre-processed at the pixel level, and the pre-processed image data is then subjected to image deep segmentation to obtain a pixel-level segmentation mask.

[0052] In step S1022, the pixel-level segmentation mask is subjected to fracture feature extraction to obtain callus features, fracture line clarity, and skeletal alignment and alignment geometry.

[0053] In step S1023, the callus features, the fracture line clarity, and the skeletal alignment and alignment geometry are used to construct an imaging feature vector.

[0054] In a specific implementation, first, the image data in the multi-modal time-series tensor can be pre-processed at the pixel level, i.e., noise suppression processing (such as non-local mean or bilateral filtering-based noise reduction) is performed on the original image at each time point, and intensity normalization is performed to eliminate gray level differences caused by different devices or scanning parameters, so as to resample voxels or pixels to unify the spatial resolution, and if it is a multi-time point image, rigid and non-rigid registration is further used to ensure consistency of the same anatomical position. Then, the pre-processed image data can be subjected to image deep segmentation, i.e., the pre-processed image data is input into a trained deep segmentation network for pixel-level segmentation. In the training stage, a joint optimization of the intersection over union loss and the boundary loss is used to improve the recognition accuracy of small callus and fracture line edges, and the segmentation inference output is a binary mask of bone tissue and callus and a pixel-level confidence map, so as to obtain a pixel-level segmentation mask that defines the region of interest for subsequent feature extraction and passes the confidence.

[0055] In addition, in specific implementation, pixel-level segmentation mask can be subjected to bone fracture feature extraction, that is, the total number of voxels (three-dimensional pixels) marked as "callus" in the pixel-level segmentation mask is directly calculated, and then multiplied by the physical volume of each voxel (calculated from the pixel spacing and layer thickness in the DICOM header file), that is, callus volume = callus voxel number x single voxel volume, so as to take the calculated callus volume as the callus feature; in the bone fracture line region of the pixel-level segmentation mask, a clear edge pixel can be extracted using an edge detection algorithm (such as Canny), and the average gradient value of the edge pixel is calculated, so as to take the average gradient value as the bone fracture line clarity, which represents the clarity of the bone fracture line; the skeleton contour segmented from the pixel-level segmentation mask is extracted, the long axis of the skeleton contour is extracted through principal component analysis or model fitting, so as to calculate the included angle of the long axes of the two ends of the bone fracture as the angulation angle, and calculate the displacement distance of the dislocation of the bone fracture ends as the displacement distance, and take the angulation angle and the displacement distance as the geometric features of the bone alignment and alignment. Finally, the radiological feature vector can be constructed based on the callus feature, the bone fracture line clarity and the geometric features of the bone alignment and alignment, that is, the feature vector composed of the callus feature, the bone fracture line clarity and the geometric features of the bone alignment and alignment is taken as the radiological feature vector.

[0056] In some embodiments, the mechanical feature extraction on the motion data in the multi-modal time series tensor can be performed in the following manner, that is:

[0057] The motion data in the multi-modal time series tensor is calibrated, and then the calibrated motion data is subjected to gait cycle detection to obtain a gait cycle sequence;

[0058] The basic gait parameters of each gait cycle in the gait cycle sequence are determined respectively;

[0059] The mechanical features and variability indexes of each gait cycle are determined according to the corresponding basic gait parameters;

[0060] The mechanical feature vector is constructed through the mechanical features and variability indexes of all gait cycles.

[0061] In specific implementation, firstly, the motion data in the multi-modal time series tensor can be calibrated, i.e., the original sensor signals are resampled at a unified sampling frequency and band-pass filtered to remove DC drift and high-frequency noise, then the gravity component is separated from the linear acceleration by using attitude filtering or complementary filtering, and the relative installation transformation between sensors is estimated by static or dynamic calibration methods, so as to output the calibrated and coordinate-aligned continuous time series signals, i.e., the calibrated motion data. Then, the calibrated motion data can be subjected to gait cycle detection, i.e., the heel strike and toe-off times are identified by peak detection and threshold judgment on the vertical acceleration or pressure signals in the motion data, and false pulses are removed by applying minimum step interval and posterior consistency verification, and a gait cycle sequence with start and end time labels is output, so as to obtain the gait cycle sequence. Secondly, the basic gait parameters of each gait cycle in the gait cycle sequence can be determined, i.e., for each gait cycle in the gait cycle sequence, the basic gait parameters can be calculated cycle by cycle, the basic gait parameters include the step time, step length, step frequency, instantaneous step speed, left and right single-limb support time and double-limb support time, and peak value and amplitude of joint angle of each cycle, the step length can be integrated based on the foot velocity and corrected by zero velocity update to reduce cumulative error, and in this way, the basic gait parameters of each gait cycle in the gait cycle sequence can be obtained.

[0062] In addition, in specific implementation, the mechanical characteristics and variability indexes of each gait cycle can be determined according to the corresponding basic gait parameters, it should be noted that in the present application, the mechanical characteristics include average speed, single-limb weight-bearing duration ratio, peak value of joint range of motion, peak value of center of gravity acceleration, and phase energy distribution, the variability indexes include coefficient of variation of step time and step length, variation amplitude of step frequency, spectral entropy and short-term frequency band energy fluctuation, and time series uncertainty measure reflecting gait stability, the above-mentioned mechanical characteristics and variability indexes can be calculated by existing algorithms, which will not be described here. Finally, the mechanical characteristic vector can be constructed by the mechanical characteristics and variability indexes of all gait cycles, i.e., the mechanical characteristics and variability indexes of all gait cycles are combined and aligned in time sequence, so as to obtain the final feature vector as the mechanical characteristic vector.

[0063] It should be noted that the deep feature extraction of the image data and the motion data in the multi-modal time sequence tensor can convert the structural change information of fracture healing and the dynamic performance of patient functional recovery from original high-dimensional data into stable, quantifiable and fusable representations, so that the image morphology, callus growth, alignment changes and factors such as gait speed, weight-bearing capacity and stability that are originally difficult to directly compare between different modalities can be uniformly expressed in the same feature space; the deep image features can accurately capture small changes such as callus formation and fracture line blurring, while the mechanical features can reflect the real movement ability and recovery trend of the patient, which helps to form a global, dynamic and quantifiable description of the nursing effect of bone injury.

[0064] In step S103, a nursing digital twin in bone injury nursing is constructed based on the imaging feature vector and the mechanical feature vector, multi-modal fusion evaluation is performed through the current state of the nursing digital twin, and a comprehensive nursing evaluation result is obtained.

[0065] In some embodiments, the construction of the nursing digital twin in bone injury nursing based on the imaging feature vector and the mechanical feature vector can be specifically performed in the following manner, that is:

[0066] The imaging feature vector and the mechanical feature vector are time-aligned and standardized to obtain a state vector in bone injury nursing;

[0067] A parameterized proxy model is constructed through the state vector;

[0068] The proxy model is individually tuned to obtain the nursing digital twin in bone injury nursing.

[0069] In a specific implementation, first, the imaging feature vector and the mechanical feature vector can be time-aligned and standardized, that is, the imaging feature vector and the mechanical feature vector are strictly aligned according to their respective time markers, and a unified normalization and standardization strategy (for example, based on training set quantile scaling or zero mean unit variance transformation) is adopted for each feature, while the quality confidence of each feature is used to weight the abnormal values and compensate for the missing values, and a state vector that is consistent in time and comparable in value is output, that is, the state vector in bone injury care. Then, a parameterized proxy model can be constructed from the state vector, that is, a parameterized proxy model is constructed with the state vector as the observation input, which consists of two parts: one is a geometry-mechanics approximation sub-model based on physics, which is used to constrain the geometric size, material stiffness and healing-related physical parameters with imaging features; the second is a data-driven dynamic sub-model, which is used to describe the boundary load, functional response and time evolution law with mechanical features, and the proxy model is expressed in the form of parameter set and exposes adjustable parameters for subsequent calibration. Finally, the proxy model can be individually parameterized, that is, individualized parameterization is implemented based on the patient's past time series observation data and the forward simulation output of the proxy model, the parameter prior can be set and the difference between the model output and the observation can be measured by the loss function, then the parameter estimation method (such as Bayesian inference or least squares optimization combined with regularization) is used to iteratively solve the posterior distribution or optimal point estimate of the parameters, and Monte Carlo sampling or robust optimization is introduced in the process to quantify the parameter uncertainty and prevent overfitting, completing the calibration of key parameters such as geometry, mechanics and healing rate; and the parameterized proxy model is coupled with its data-driven dynamic part, and it is deployed as a runnable care digital twin, so that the care digital twin in bone injury care can be obtained.

[0070] In some embodiments, the multi-modal fusion evaluation by the current state of the care digital twin is to input the current state of the care digital twin as input data into a multi-modal evaluation model for evaluation, and then obtain a comprehensive care evaluation result.

[0071] In a specific implementation, first, the current state of the nursing digital twin can be provided as input data to the multi-modal evaluation model, i.e., normalization and confidence weighting are performed on the current state to ensure the comparability of features from different sources in terms of numerical scale and reliability, and the processed vector is directly input to the next step. Then, a multi-modal feature encoder is used to encode the imaging sub-vector and the mechanical sub-vector of the nursing digital twin, respectively. The encoder can be a network based on sequence modeling or a transformer based on self-attention mechanism, which outputs the imaging time sequence representation and the mechanical time sequence representation, respectively. The encoder also returns the attention weight of each modality as intermediate explanatory information. Second, the time sequence representation and the corresponding attention weight of each modality are input into a fusion unit. The fusion unit performs weighted aggregation based on the attention weight and the confidence, and combines a time consistency correction module to eliminate the influence of short-term abnormal windows. The fusion unit outputs a unified multi-modal comprehensive representation. Finally, the multi-modal comprehensive representation can be input into the evaluation decision layer, which performs regression and classification tasks in parallel to output the comprehensive nursing evaluation results, including functional recovery score, pain relief score, life ability score, and comprehensive nursing score. During the evaluation process, a prediction method with uncertainty estimation is used to output point estimates and confidence intervals simultaneously.

[0072] It should be noted that the imaging feature vector and the mechanical feature vector are used together to construct the nursing digital twin, and multi-modal fusion evaluation is carried out based on the current state of the digital twin. This can integrate complementary information with a unified parameterized physiological structure-function model, so that the model can reflect structural changes such as callus formation and alignment from the imaging perspective, and can also depict functional changes such as gait recovery and load capacity from the mechanical perspective. On this basis, multi-modal fusion evaluation can accurately quantify the contribution of each modality feature to the evaluation result through attention weight and sensitivity analysis, thereby significantly improving the accuracy and interpretability of the comprehensive evaluation result.

[0073] In step S104, an imaging contribution heat map and a mechanical contribution heat map are constructed based on the comprehensive nursing evaluation result, and the nursing digital twin in bone injury nursing is dynamically updated based on the imaging contribution heat map and the mechanical contribution heat map.

[0074] In some embodiments, the imaging contribution heat map and the mechanical contribution heat map can be constructed based on the comprehensive nursing evaluation result in the following manner, i.e.:

[0075] Obtaining intermediate output in multi-modal fusion evaluation;

[0076] Extracting an influence quantity for explanation from the intermediate output and the comprehensive nursing evaluation result;

[0077] performing reverse mapping on the image-related influence quantity in the influence quantity to obtain a pixel contribution map;

[0078] performing regional aggregation on the pixel contribution map to further generate an imaging contribution heat map;

[0079] extracting a contribution value sequence of a mechanical feature vector in the influence quantity, and performing rule projection on the contribution value sequence to obtain a mechanical contribution distribution map;

[0080] performing time smoothing and confidence weighting processing on the mechanical contribution distribution map to further generate a mechanical contribution heat map.

[0081] In specific implementation, first, intermediate outputs and final scores can be obtained from a multi-modal fusion evaluation process as original inputs. Then, influence quantities for explanation can be extracted from the intermediate outputs and comprehensive nursing evaluation results, including model attention weights on input features, gradient sensitivity of prediction scores on input pixels or features, and sensitivity changes obtained based on input perturbation, which serve as the basis for subsequent mapping and projection. Second, image-related influence quantities in the influence quantity can be reversely mapped, specifically, pixel-level gradient sensitivity and model output attention weights are multiplied according to pixel correspondence and combined with pixel confidence in the segmentation mask for weighting to obtain an original pixel contribution map, and local statistical filtering is performed on the original pixel contribution map to suppress noise and fill in isolated artifacts, and the output is a smoothed pixel contribution map with confidence annotation. Further, regional aggregation can be performed on the pixel contribution map, that is, the pixel contribution map is aggregated in regions with anatomical partitions or segmentation masks as units, the cumulative contribution value of each region is calculated and normalized to form imaging contribution heat map layers layered by anatomical sites and time points, thereby obtaining an imaging contribution heat map.

[0082] In addition, in specific implementation, a contribution value sequence of a mechanical feature vector can be extracted from the influence quantity, that is, a contribution value sequence of the mechanical feature vector over each time window or gait cycle is extracted from the influence quantity, and the contribution value sequence is projected onto space or gait phase according to a pre-defined mapping rule, specifically including projecting left-right support time ratio to corresponding parts of left and right lower limbs, projecting joint range of motion to corresponding joint regions, projecting step frequency and step speed to the phase axis of gait cycle, to generate an original mechanical contribution distribution map. Finally, time smoothing and confidence weighting processing can be performed on the mechanical contribution distribution map, that is, smoothing processing is performed on the mechanical contribution distribution map along the time axis to weaken short-term abnormalities, and sensor quality metrics and event recognition confidence are used to weight and adjust the contribution values at each time point to reduce the impact of low-quality data on visualization, and the output is a time-continuous mechanical contribution heat map with confidence.

[0083] In some embodiments, dynamically updating the care digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map can be implemented in the following manner, that is:

[0084] Inputting the imaging contribution heat map and the mechanical contribution heat map as external feedback signals into the care digital twin;

[0085] Adjusting internal parameters of the care digital twin according to the external feedback signals, thereby obtaining an updated care digital twin.

[0086] In specific implementation, first, the imaging contribution heat map and the mechanical contribution heat map can be inputted as external feedback signals into the care digital twin. Then, the care digital twin adjusts internal parameters according to the feedback signals, which specifically includes two types of operations: one type is to increase the weight or reduce the prior constraint of the corresponding bone geometry parameters, callus volume estimation, fracture line clarity index and related healing rate parameters for the region with high imaging contribution, so as to improve the sensitivity of the care digital twin to key structural changes; the other type is to dynamically adjust the gait cycle parameters, step frequency, step speed, support time ratio and joint range of motion and other related parameters for the feature with high mechanical contribution, while increasing the uncertainty or regularization constraint for low contribution or unstable features to reduce their noise interference; in the adjustment process, the care digital twin iteratively calculates the new estimated value of each parameter through optimization algorithm or Bayesian update method, to ensure that the adjusted parameters can reflect the latest contribution information and maintain the stability of the overall care digital twin; finally, the parameter set after adjustment is coupled with the dynamic prediction module of the twin to form an updated care digital twin, which can more accurately perform multi-modal fusion evaluation, generate explanatory contribution heat map and support closed-loop update mechanism to continuously optimize the accuracy and interpretability of bone injury care effect evaluation when new imaging or mechanical state vectors are inputted subsequently.

[0087] It should be noted that, according to the construction of the imaging contribution heat map and the mechanical contribution heat map based on the comprehensive nursing evaluation result, and the dynamic updating of the nursing digital twin by using the two types of heat maps, the key influencing factors in the comprehensive evaluation result can be explicitly visualized and directly act on the internal parameters of the nursing digital twin, so that the sensitivity and accuracy of the digital twin to the structural healing indicators and functional mechanical characteristics are optimized. Specifically, the imaging heat map highlights the contribution of structural key regions such as callus volume, fracture line clarity and bone alignment to the evaluation result, while the mechanical heat map reveals the influence of functional key characteristics such as gait parameters, joint range of motion and support time. These information as external feedback guide the twin to dynamically adjust parameters and weights, realize the enhancement of high contribution characteristics and the uncertainty weighting of low contribution characteristics, thus forming a closed-loop updating mechanism. Through the updating strategy based on contribution visualization, the nursing digital twin can provide interpretable evaluation output, thereby significantly improving the accuracy and interpretability of bone injury nursing effect evaluation.

[0088] As can be seen from the above, in the present application, first, the image data and motion data in the multi-modal time series tensor are respectively subjected to deep feature extraction, which can convert the structural change information of fracture healing and the dynamic performance of patient functional recovery from original high-dimensional data into stable, quantifiable and fusable representations; then, the imaging feature vector and the mechanical feature vector are used together to construct a nursing digital twin, and based on the current state of the twin, a multi-modal fusion evaluation is carried out, which can integrate complementary information in a unified parameterized physiological structure-function model, thereby significantly improving the accuracy of the comprehensive evaluation result; finally, according to the construction of the imaging contribution heat map and the mechanical contribution heat map based on the comprehensive nursing evaluation result, and the dynamic updating of the nursing digital twin by using the two types of heat maps, the key influencing factors in the comprehensive evaluation result can be explicitly visualized and directly act on the internal parameters of the nursing digital twin, so that the sensitivity and accuracy of the digital twin to the structural healing indicators and functional mechanical characteristics are optimized, and through the updating strategy based on contribution visualization, the nursing digital twin can provide interpretable evaluation output, thereby significantly improving the accuracy and interpretability of bone injury nursing effect evaluation.

[0089] In summary, the technical scheme adopted by the present application can construct a nursing digital twin with interpretable output and closed-loop updating mechanism based on contribution visualization, to improve the interpretability and accuracy of bone injury nursing effect evaluation.

[0090] In addition, another aspect of the present application, in some embodiments, the present application provides a bone injury nursing effect evaluation system based on multi-modal data fusion, referring to Figure 3 The figure is a structural schematic diagram of a bone injury nursing effect evaluation system based on multi-modal data fusion according to some embodiments of the present application, which comprises:

[0091] The data acquisition module 201 is configured to acquire multi-modal data in bone injury care, pre-process the multi-modal data, and obtain a multi-modal time series tensor.

[0092] The feature extraction module 202 is configured to perform deep feature extraction on image data in the multi-modal time series tensor, to obtain an imaging feature vector, and perform mechanical feature extraction on motion data in the multi-modal time series tensor, to obtain a mechanical feature vector.

[0093] The fusion evaluation module 203 is configured to construct a care digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector, perform multi-modal fusion evaluation through a current state of the care digital twin, and obtain a comprehensive care evaluation result.

[0094] The dynamic updating module 204 is configured to construct an imaging contribution heat map and a mechanical contribution heat map according to the comprehensive care evaluation result, respectively, and dynamically update the care digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map.

[0095] In addition, the present application also provides a computer device, which comprises a memory and a processor, the memory stores code, and the processor is configured to acquire the code and execute the above-mentioned bone injury care effect evaluation method based on multi-modal data fusion.

[0096] In some embodiments, with reference to Figure 4 The figure is a structural schematic diagram of a computer device for implementing the bone injury care effect evaluation method based on multi-modal data fusion according to some embodiments of the present application. The bone injury care effect evaluation method based on multi-modal data fusion in the above-mentioned embodiments can be implemented by the computer device shown in the figure, which comprises at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304. Figure 4

[0097] The processor 301 can be a general central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more circuits for controlling the execution of the bone injury care effect evaluation method based on multi-modal data fusion in the present application.

[0098] The communication bus 302 can be used to transmit information between the above-mentioned components.

[0099] ​The memory 303 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magneto-optical disk, a magnetic disk or other magnetic storage device, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 303 can exist independently, and is connected to the processor 301 through the communication bus 302. The memory 303 can also be integrated with the processor 301.

[0100] The memory 303 is configured to store program codes for implementing the solutions of the present application, and the processor 301 is configured to control the execution of the program codes. The processor 301 is configured to execute the program codes stored in the memory 303. The program codes can include one or more software modules. The determination of the bone injury nursing effect evaluation method based on multi-modal data fusion in the above embodiments can be implemented by one or more software modules in the program codes of the processor 301 and the memory 303.

[0101] The communication interface 304 is configured to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., using any transceiver-like mechanism.

[0102] In specific implementations, as an example, the computer device can include multiple processors, each of which can be a single-CPU processor or a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0103] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer device.

[0104] In addition, the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the bone injury nursing effect evaluation method based on multi-modal data fusion.

[0105] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application.

[0106] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for evaluating the effect of bone injury care based on multi-modal data fusion, characterized in that, The method comprises the following steps: obtaining multi-modal data in bone injury care, pre-processing the multi-modal data to obtain a multi-modal time series tensor; extracting deep features from image data in the multi-modal time series tensor to obtain an imaging feature vector, and extracting mechanical features from motion data in the multi-modal time series tensor to obtain a mechanical feature vector; constructing a nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector, performing multi-modal fusion evaluation through a current state of the nursing digital twin to obtain a comprehensive nursing evaluation result; constructing an imaging contribution heat map and a mechanical contribution heat map according to the comprehensive nursing evaluation result, and dynamically updating the nursing digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map; wherein constructing the imaging contribution heat map and the mechanical contribution heat map according to the comprehensive nursing evaluation result specifically comprises: obtaining an intermediate output in multi-modal fusion evaluation; extracting an influence quantity for explanation from the intermediate output and the comprehensive nursing evaluation result; performing reverse mapping on image-related influence quantities in the influence quantity to obtain a pixel contribution map; performing regional aggregation on the pixel contribution map to generate an imaging contribution heat map; extracting a contribution value sequence of the mechanical feature vector from the influence quantity to generate a mechanical contribution distribution map by performing rule projection on the contribution value sequence; and performing time smoothing and confidence weighting processing on the mechanical contribution distribution map to generate a mechanical contribution heat map; wherein dynamically updating the nursing digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map specifically comprises: inputting the imaging contribution heat map and the mechanical contribution heat map as external feedback signals into the nursing digital twin; adjusting internal parameters of the nursing digital twin according to the external feedback signals to obtain an updated nursing digital twin.

2. The bone injury care effect evaluation method based on multi-modal data fusion according to claim 1, characterized in that, The multi-modal data comprises image data and motion data in bone injury care. 3.The bone injury nursing effect evaluation method based on multi-modal data fusion of claim 1, wherein, The pre-processing of the multi-modal data comprises timestamp alignment and missing value filling of the multi-modal data to obtain a multi-modal time series tensor. 4.The bone injury nursing effect evaluation method based on multi-modal data fusion of claim 1, wherein, The deep feature extraction from the image data in the multi-modal time series tensor to obtain an imaging feature vector specifically comprises: pixel pre-processing of the image data in the multi-modal time series tensor, and then image depth segmentation of the pixel pre-processed image data to obtain a pixel-level segmentation mask; fracture feature extraction from the pixel-level segmentation mask to obtain callus features, fracture line clarity, and skeletal alignment and alignment geometry features; constructing the imaging feature vector based on the callus features, the fracture line clarity, and the skeletal alignment and alignment geometry features.

5. The method for evaluating the effect of bone injury care based on multi-modal data fusion according to claim 1, wherein, The mechanical feature extraction from the motion data in the multi-modal time series tensor to obtain a mechanical feature vector specifically comprises: calibration of the motion data in the multi-modal time series tensor, and then gait cycle detection of the calibrated motion data to obtain a gait cycle sequence; determining basic gait parameters of each gait cycle in the gait cycle sequence; Determine the mechanical characteristics and variability indexes of each gait cycle according to the corresponding basic gait parameters; Construct the mechanical characteristic vector through the mechanical characteristics and variability indexes of all gait cycles.

6. The method for evaluating the effect of bone injury care based on multi-modal data fusion according to claim 1, wherein, Constructing the nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector specifically includes: Time-aligning and standardizing the imaging feature vector and the mechanical feature vector to obtain a state vector in bone injury care; Constructing a parameterized proxy model through the state vector; Individualizing the proxy model to obtain the nursing digital twin in bone injury care.

7. The method for evaluating the effect of bone injury care based on multi-modal data fusion according to claim 1, wherein, The multi-modal fusion evaluation through the current state of the nursing digital twin is to input the current state of the nursing digital twin as input data into a multi-modal evaluation model for evaluation, and then obtain a comprehensive nursing evaluation result.

8. A bone injury nursing effect evaluation system based on multi-modal data fusion, used for executing a bone injury nursing effect evaluation method based on multi-modal data fusion according to any one of claims 1 to 7, characterized in that, It includes: A data acquisition module is configured to acquire multi-modal data in bone injury care, pre-process the multi-modal data, and obtain a multi-modal time series tensor; A feature extraction module is configured to extract deep features from image data in the multi-modal time series tensor to obtain an imaging feature vector, and extract mechanical features from motion data in the multi-modal time series tensor to obtain a mechanical feature vector; A fusion evaluation module is configured to construct a nursing digital twin in bone injury care based on the imaging feature vector and the mechanical feature vector, and to perform multi-modal fusion evaluation through the current state of the nursing digital twin to obtain a comprehensive nursing evaluation result; A dynamic updating module is configured to construct an imaging contribution heat map and a mechanical contribution heat map based on the comprehensive nursing evaluation result, and to dynamically update the nursing digital twin in bone injury care based on the imaging contribution heat map and the mechanical contribution heat map.

9. A computer device, comprising: The computer device includes a memory and a processor, the memory stores code, the processor is configured to acquire the code, and execute the bone injury care effect evaluation method based on multi-modal data fusion as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the bone injury care effect evaluation method based on multi-modal data fusion as claimed in any one of claims 1 to 7.

Images