Medical health monitoring method based on multi-modal data

Abstract

The invention discloses a medical health monitoring method based on multi-modal data, and relates to the technical field of health monitoring, and the method comprises the steps: collecting the multi-modal data, analyzing the difference of different modal data point pairs, defining the matching cost, calculating an accumulated cost matrix, and carrying out the dynamic alignment of a time sequence. And carrying out isolated scoring and abnormal value elimination by adopting an isolated forest algorithm, and constructing an observation tensor according to the modal number, the time point number and the modal characteristics of the multi-modal data. According to the method, multi-dimensional expansion of modal features is realized through observation tensor construction, multi-modal data is expanded from a time sequence to a tensor space, time features of single modals are integrated, interactivity between the modals is reserved, and through iterative updating of interaction weights, the overall interaction contribution of the modal features is optimized and corrected, so that the multi-dimensional expansion of the modal features is realized. The modal interaction weight is updated through iterative propagation, the initial weight of the modal interaction weight is considered, and other modals and the interaction strength of the modal interaction weight are combined.

Description

Technical Field

[0001] This invention relates to the field of health monitoring technology, and in particular to a medical health monitoring method based on multimodal data. Background Technology

[0002] With the rapid development of medical and information technologies, multimodal data-based medical and health monitoring technologies are increasingly becoming an important direction for medical diagnosis and personalized health management. Multimodal data refers to physiological signals, behavioral data, and environmental information collected through various sensors or monitoring devices, including but not limited to heart rate (HR), blood oxygen saturation (SpO2), heart rate variability (HRV), and blood glucose levels. This modal data can provide rich information for assessing human health status. Due to differences in sampling frequency, dynamic characteristics, temporal synchronization, and numerical range among different modal data, these data exhibit heterogeneity, imbalance, and spatiotemporal mismatch. In current technologies, research on temporal alignment and fusion modeling of multimodal data is a core issue. Through alignment and collaborative analysis, the interaction relationships between modalities can be captured, and potential health trends can be uncovered.

[0003] However, existing technologies generally employ simple fixed-time alignment or modal features selected based on human experience, which fails to accurately describe the complex dynamic interactions between modalities. Common feature analysis or risk assessment methods are limited to the independent analysis of single-modal data or multiple modalities, lacking the characterization of dynamic correlations between multimodal data characteristics, especially lacking the quantification of the intermodal interaction dependencies in time series. They fail to flexibly handle multimodal data with large volumes, complex features, and dynamic changes, resulting in insufficient accuracy in outlier detection and removal. At the same time, existing technologies tend to ignore the correlations between modal features, making it difficult for the completion results to truly reflect the health status. Furthermore, for dimensionality reduction of high-dimensional multimodal features and the calculation of risk indices, existing technologies typically use principal component analysis or simple linear dimensionality reduction methods, failing to effectively consider the imbalance of modal features and their contribution to health assessment indicators. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a medical and health monitoring method based on multimodal data to address the lack of quantification of the intermodal interaction dependencies in time series data, the inability to flexibly handle large amounts of data, complex features, and dynamic changes in multimodal data, resulting in insufficient accuracy in outlier detection and removal. At the same time, existing technologies tend to ignore the correlation between modal features, making it difficult for the completion results to truly reflect the health status. Furthermore, for dimensionality reduction of high-dimensional multimodal features and calculation of risk indices, existing technologies usually use principal component analysis or simple linear dimensionality reduction methods, which fail to effectively consider the imbalance of modal features and their contribution to health assessment indicators.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a medical health monitoring method based on multimodal data, comprising: Multimodal data is collected, the differences between data point pairs of different modalities are analyzed, the matching cost is defined and the cumulative cost matrix is ​​calculated, dynamic alignment of time series is performed, isolation scoring is performed and outlier removal is performed using the isolation forest algorithm, observation tensors are constructed based on the number of modalities, number of time points and modal characteristics of multimodal data, and the nuclear norm is used as a low-rank constraint term to construct the completion objective function to complete missing values; Calculate the dynamic offset between modal features and use it as the feature difference value to construct a dynamic modal dependency matrix. Combine the cross-correlation between modalities to calculate the interaction weight in the dynamic modal dependency matrix. Optimize and correct the overall interaction contribution of each modal feature to the tensor as the optimized modal weight. Then, perform weighted fusion on each time slice of the multimodal tensor to generate optimized overall feature values. Finally, perform secondary propagation to determine the final optimized weight. Based on the overall feature value, the modality is split, and a Gaussian kernel function is introduced to calculate the similarity of the weighted modal features of different time point pairs, and feature decomposition is performed to extract low-dimensional embedding features; Calculate the health risk index, assess the health risks of different modalities at different time points, mark the risks, and count the number of occurrences to generate health monitoring alerts.

[0007] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the following steps are included: defining a matching cost and calculating a cumulative cost matrix; performing dynamic time series alignment; employing an isolated forest algorithm for isolated scoring and outlier removal; constructing an observation tensor based on the number of modalities, number of time points, and modal characteristics of the multimodal data; using the nuclear norm as a low-rank constraint term to construct a completion objective function; and completing missing values, including... For time series data of different modalities collected, dynamic time alignment is performed. Euclidean distance is used to describe the difference between each pair of points in the modal data at different times, and it is used as the matching cost function. The cumulative cost matrix is ​​calculated using a dynamic programming algorithm, which represents the minimum cost required to align the sequence from its starting point to the desired match. Based on the final cumulative cost matrix, the aligned time trajectory is found using a backtracking method, and dynamic alignment of the time series is performed. Outlier detection is performed on the data points of each modality. The isolated forest algorithm is used to construct isolated trees and perform isolation scoring to evaluate the ease with which data points are separated in the tree. Anomaly thresholds are determined based on historical experience. If an isolated score is less than or equal to the anomaly threshold, the corresponding data point is judged to be an outlier and removed. Based on the filtered modal time series data, an observation tensor is constructed according to the number of modes, the number of time points, and modal characteristics; Calculate the Frobenius norm of the observed Zhang data, and use the nuclear norm as a low-rank constraint term to construct the completion objective function to complete the missing values; Alternating Direction Multiplier Method (ADMM) is used for target optimization, and an auxiliary tensor is introduced. This represents the low-rank approximation update objective function, and constructs the Lagrange function; The mask tensor is defined and iteratively optimized based on the Alternating Direction Multiplier Method (ADMM). The complete data tensor to be optimized is updated by element-wise multiplication based on the update objective function. Perform a soft thresholding operation on the singular value diagonal matrix, use the thresholded singular values ​​to generate a reconstructed low-rank tensor, and update the Lagrange multipliers. The complete data tensor obtained during convergence optimization is used as the supplementary data tensor.

[0008] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the optimization correction of the modal features of each modality to the overall interaction contribution of the tensor is used as the optimized modal weight, and the multimodal tensor is weighted and fused for each time slice to generate optimized overall feature values, and a second propagation is performed to determine the final optimized weights, including: The dynamic characteristic dependencies between modes are analyzed, and the dynamic offset between modal features is calculated at each time point to quantify the instantaneous correlation between modes as the feature difference value. The feature difference values ​​are normalized and the full-time average correlation between modes is obtained. The modal dependency values ​​are analyzed to construct a dynamic modal dependency matrix. Based on the dynamic modal dependency matrix, the confidence of outliers removed by isolated scores is used as the initial interaction weight. The interaction weight of a modality is set as a combination of the weighted propagation of the interaction strength of other modalities and its own interaction weight. Iterative updates are performed. The iteration stops when the change value of the modal weight to be updated is less than or equal to the change threshold. The modal features of each modality are optimized and corrected to the overall interaction contribution of the tensor, which is used as the optimized modal weight. Based on the optimized modal weights, the multimodal tensor is weighted and fused for each time slice to generate an optimized dynamic fusion feature tensor as the overall feature value. The overall feature value is then fed back to influence the modal weights, and a second propagation is performed to further optimize the modal weights.

[0009] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the method involves: splitting the overall feature value according to the modality, introducing a Gaussian kernel function to calculate the similarity of weighted modal features of different time point pairs, and performing feature decomposition to extract low-dimensional embedded features, including... Based on the overall feature value, the modality is split, and a Gaussian kernel function is introduced in combination with the final optimized weight to calculate the similarity of the weighted modal features of different time point pairs; The kernel function is centered and eigenvalue decomposition is performed to extract low-dimensional embedding features.

[0010] As a preferred embodiment of the multimodal data-based medical health monitoring method of the present invention, the step of calculating a health risk index and assessing the health risk of different modalities at each time point includes: Based on the extracted low-dimensional embedding feature data, a health risk index is defined for each time point t. The health risk index is calculated by combining the low-dimensional embedding features and modality hierarchies to assess the health risk of different modalities at each time point.

[0011] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the risk labeling includes: The health safety threshold is based on the sum of the mean and two standard deviations of the historical health risk index. If the calculated health risk value is greater than or equal to the health and safety threshold, then a health risk is determined to exist for modal time, and the risk is marked.

[0012] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the step of generating health monitoring alerts by counting the number of occurrences includes: The system counts the number of times a risk marker appears within a continuous time window and sets an alarm threshold based on historical experience. If the number of triggers is greater than or equal to the alarm threshold, a health monitoring alarm is triggered.

[0013] As a preferred embodiment of the medical and health monitoring method based on multimodal data described in this invention, the method includes: collecting multimodal data, including... Multimodal data acquisition is performed using physiological sensors, including heart rate data, blood oxygen saturation (SpO2) data, and heart rate variability (HRV) data.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the medical health monitoring method based on multimodal data as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the medical and health monitoring method based on multimodal data as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By constructing observation tensors, multidimensional expansion of modal features is achieved, extending multimodal data from time series to tensor space. This not only integrates the temporal features of single modalities but also preserves the interactivity between modalities. Through iterative updates of interaction weights, the overall interaction contribution of modal features is optimized and corrected. Modal interaction weights are updated through iterative propagation, taking into account not only their initial weights but also the interaction strength between them and other modalities. The feedback from the dynamically fused feature tensor influences the modal weights for secondary propagation, further optimizing the modal contribution to the fusion result. Finally, a health-related low-dimensional representation is generated through the embedded feature space, directly reflecting the core health status after dynamic fusion of modal characteristics. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the medical and health monitoring method based on multimodal data in Example 1.

[0019] Figure 2 This is a schematic diagram of the missing value completion process of the medical and health monitoring method based on multimodal data in Example 1. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, referring to Figures 1 to 2 This is the first embodiment of the present invention, which provides a medical health monitoring method based on multimodal data, including the following steps: S1. Collect multimodal data, analyze the differences between data point pairs of different modalities, define the matching cost and calculate the cumulative cost matrix, perform dynamic alignment of time series, use the isolation forest algorithm to perform isolation scoring and outlier removal, construct the observation tensor based on the number of modalities, number of time points and modal characteristics of multimodal data, use the nuclear norm as a low-rank constraint term to construct the completion objective function, and complete the missing values. Preferably, the number of times the risk marker appears within a continuous time window is counted, and an alarm threshold is set based on the number of triggers in historical experience. If the number of triggers is greater than or equal to the alarm threshold, a health monitoring alarm is issued.

[0024] Preferably, the matching cost is defined and the cumulative cost matrix is ​​calculated. Dynamic alignment of the time series is performed. The isolation forest algorithm is used for isolation scoring and outlier removal. An observation tensor is constructed based on the number of modalities, the number of time points, and modal features of the multimodal data. The nuclear norm is used as a low-rank constraint term to construct the completion objective function, and missing values ​​are imputed, including... For the collected time series data of different modalities, dynamic time alignment is performed. The difference between each pair of data points at different times is described using Euclidean distance, which is used as the matching cost function, expressed as: ; in, express and The cost of matching time sequences i and j. This represents the sampled value at the i-th time point of the time series data of the Hr mode. Indicates the first The sampled value at the j-th time point of the modal time series data; The cumulative cost matrix is ​​calculated using a dynamic programming algorithm, representing the minimum matching cost required to align from the start of the sequence, and is expressed as: ; in, Let represent the minimum cumulative matching cost of the (i,j)th time point pair in the cumulative cost matrix. This indicates selecting the minimum cumulative cost. This indicates vertical alignment, considering only the first... Modal time series, This indicates horizontal alignment, considering only the time series of the Hr mode. This indicates diagonal alignment, while considering data from both modalities. Based on the final cumulative cost matrix, the aligned time trajectory is found using backtracking, and dynamic alignment of the time series is performed, as shown below:

[0025] in, Representing modes The optimal matching trajectory from time point i to modality Hr time point j is a dynamically established mapping achieved by finding the optimal time alignment relationship that minimizes the cost difference. Indicates the total number of time points. Representing modes Time point i and mode Hr at The matching value of the mapping time points, Indicates the first Modal time series data in The sampled value of the mapping time; Outlier detection is performed on the data points for each modality. The Isolation Forest algorithm is used to construct an isolation tree and perform isolation scoring to evaluate the ease with which a data point can be separated in the tree, as shown below: ; in, This represents the isolated score value of data point x. This represents the path length of data point x in the isolated tree (the number of splits from the root node to the terminal node). This path length indicates the position of data point x in the corresponding modal time series data. The expected value of the tree depth in the isolated forest is automatically obtained during training, and x represents a data point in the time series. Anomaly thresholds are determined based on historical experience. If an isolated score is less than or equal to the anomaly threshold, the corresponding data point is judged to be an outlier and removed. Based on the removed modal time series data, an observation tensor is constructed according to the number of modalities, the number of time points, and modal features. The modal features are defined based on heart rate data, blood oxygen saturation (SpO2) data, and heart rate variability (HRV) data. Specifically, the modal features for heart rate data include the mean heart rate value, standard deviation of heart rate, and rate of heart rate rise; the modal features for blood oxygen saturation data include the mean blood oxygen value and rate of blood oxygen fall; and the modal features for heart rate variability data include the standard deviation and mean time interval of the time interval between heartbeats (RR interval). These are expressed as follows: ; in, Let L represent the multimodal observation tensor, T represent the total number of modes, F represent the total number of time points, and F represent the total number of modal features. Represents a dataset; Calculate the Frobenius norm of the observed tens of data, and use the nuclear norm as a low-rank constraint term to construct the completion objective function for missing values, expressed as: ; in, This indicates that the objective function needs to be completed. This represents the constraint used to constrain the observed data, ensuring that the completed tensor is consistent with the original observations. It is the square of the Frobenius norm, guaranteeing that the completed tensor X is as close as possible to the original tensor on the known observed data. , This represents the nuclear norm constraint, used to ensure the low-rank property of completion, where This represents a regularization hyperparameter that controls the balance between data consistency and sparsity terms. It can be preset through cross-validation or experiments. This represents the complete data tensor that needs to be optimized. Alternating Direction Multiplier Method (ADMM) is used for objective optimization. An auxiliary tensor is introduced to represent the low-rank approximation of the objective function for updating, and a Lagrangian function is constructed, expressed as:

[0026] in, This indicates that the objective function is being updated. express The low-rank approximation auxiliary tensor in the update objective function , Represents the value of the Lagrange function. The step size parameter is set based on historical experience, and U represents the Lagrange multiplier. Define the mask tensor Mark the observation positions (1 indicates data is present, 0 indicates missing data), perform iterative optimization based on the Alternating Direction Multiplier Method (ADMM), and update the complete data tensor to be optimized by element-wise multiplication based on the update objective function, as follows:

[0027] in This represents element-wise multiplication. This represents the complete data tensor that needs to be optimized for updating; right Singular value decomposition is performed into a diagonal matrix of singular values, a left singular vector, and the transpose of the right singular vector. The low-rank approximate auxiliary tensor is updated based on the singular value soft thresholding (SVT) method for updating the objective function, expressed as:

[0028] in, Represents a singular value diagonal matrix. Represents a left singular vector. This represents the calculation of the transpose of a right singular vector; Applying a soft thresholding operation to a singular value diagonal matrix is ​​expressed as: ; in, This represents the singular value matrix after soft thresholding. Represents the regularization parameter. This represents the penalty coefficient, determined based on experimental data; Reconstruct the low-rank tensor using thresholded singular value generation. And update the Lagrange multipliers , is represented as:

[0029] The complete data tensor obtained based on the need for updating needs With reconstructing low-rank tensors The difference is used to determine the convergence difference. If the convergence difference is less than or equal to the historical convergence threshold, the iteration is stopped. The complete data tensor obtained during convergence optimization is used as the supplementary data tensor.

[0030] By using time alignment and matching cost matrix, different modal sampling frequencies or time span misalignments can be synchronized, constructing a unified time reference. This eliminates the asynchronous problem in the time series alignment process and effectively avoids information loss or interference caused by different frequencies. Especially when there are different physiological signal time delays between modalities (such as between heart rate and blood oxygen saturation signals), time alignment improves the ability to accurately reflect the interaction characteristics between modalities. Robustness of data cleaning is achieved through outlier removal using Isolation Forests. This method not only removes invalid points but also ensures that the remaining data retains its micro-distribution characteristics. Combined with the time-point mapping of dynamically aligned outputs, the cleaned data provides high-quality input for subsequent fine-grained characterization of modal features, rather than causing overall distortion after simply removing outliers. By constructing observation tensors to achieve multidimensional expansion of modal features, multimodal data is extended from time series to tensor space. This not only integrates the temporal features of single modalities but also preserves the interactivity between modalities. Tensor construction explicitly combines the multi-feature structures of modalities (such as heart rate and blood oxygen), avoiding the loss of interactive information caused by independent analysis of two-dimensional features. In addition, by measuring the distribution characteristics of the data through tensor norms, the recovery range of missing values ​​can be accurately constrained during the completion stage, thereby improving the realism of data completion. Data integrity is enhanced through tensor completion constrained by the nuclear norm. The introduction of the nuclear norm addresses the completion distortion problem caused by sparse distribution or modal information conflict in multimodal data. For example, when the standard deviation of heart rate and the standard deviation of HRV time interval are highly correlated, traditional linear interpolation may ignore this correlation. However, the nuclear norm constraint ensures that the global low-rank characteristics of the data are preserved during the completion process. The convergence of low-rank completion is achieved through dynamic tensor iterative optimization. The advantage of the ADMM method lies in its ability to decompose complex optimization problems, transforming the update of the completed tensor into subproblems to be solved step by step. By combining singular value decomposition and soft thresholding, the dominant role of different modes and features in the tensor is reflected. Excessively large singular values ​​correspond to the main structure of the data, while excessively small singular values ​​reflect noise or sparsity characteristics, which have physiological significance in real data.

[0031] S2, calculate the dynamic offset between modal features, construct the dynamic modal dependency matrix as the feature difference value, calculate the interaction weight in the dynamic modal dependency matrix by combining the cross-correlation between modalities, optimize and correct the overall interaction contribution of each modal feature to the tensor as the optimized modal weight, and perform weighted fusion on each time slice of the multimodal tensor to generate the optimized overall feature value, and perform secondary propagation to determine the final optimized weight; Preferably, the modal features of each modality are optimized and corrected to contribute to the overall interaction of the tensor as optimized modal weights. Weighted fusion is then performed on each time slice of the multimodal tensor to generate optimized overall feature values. A second propagation is then performed to determine the final optimized weights, including... The dynamic characteristic dependencies between modes are analyzed, and the dynamic offset between modal features is calculated at each time point. The instantaneous correlation between modes is quantified as the feature difference value, expressed as: ; in, This represents the characteristic difference value between mode b and mode h at time t. This represents the eigenvalue of the f-th mode at time point t in the completed data tensor of mode b. This represents the time-shifted data value of mode j in the completed data tensor, where f is the modal feature index. Indicates the time offset. Indicates the best matching trajectory The alignment delay between mode b and mode h in the time series is obtained; Normalize the feature difference values ​​and obtain the full-time average correlation between modes. Analyze the mode dependency values ​​to construct a dynamic mode dependency matrix, which is expressed as:

[0032] in, This represents the dynamic correlation between mode b and mode h. Represents all modes The sum of eigenvalues ​​at time point t is used to normalize the characteristic offset. This represents the dependence of mode b on mode h. A dynamic mode dependency matrix is ​​constructed based on the dependencies of different modes. This represents the dynamic correlation between mode b and mode k, where L represents the total number of modes; Based on the dynamic modality dependency matrix, confidence scores calculated from outliers removed by isolated ratings are used as initial interaction weights. The interaction weights of each modality are defined as a combination of the weighted propagation of the interaction strengths of other modalities and the modality's own interaction weights. Iterative updates are performed until the change in the modality weights to be updated is less than or equal to a change threshold (determined based on the mean and standard deviation of historical change thresholds). The modality features of each modality are then optimized to compensate for their overall interaction contribution to the tensor, resulting in optimized modality weights, expressed as:

[0033] in, This represents the initial weights of mode b. This represents the total number of data points for mode b. This represents the number of data points that were detected as abnormal in mode b. and Let these represent the weights of mode b in the (k+1)th and kth iterations, respectively. This represents the dynamic adjustment factor, determined based on the sum of the mean and standard deviation of the historical adjustment factors. This represents the weight of mode h in the k-th iteration; Based on the optimized modal weights, the multimodal tensors are weighted and fused at each time slice to generate an optimized dynamic fusion feature tensor as the overall feature value. This overall feature value then feeds back into the modal weights, and a second propagation is performed to further optimize the modal weights, as shown below:

[0034] in, This represents the overall feature value of the fused time-t modal features f. This represents the optimized mode weights of mode b. This represents the final optimized weights for mode b.

[0035] By calculating the dynamic offset between modal features, the instantaneous correlation between modalities is quantified as the feature difference value. The time offset obtained by the best matching trajectory alignment can be used to explicitly adjust the time point reflection relationship of different modalities, making the offset calculation more dynamic feature correlation. The average correlation between modes over the entire time is obtained by normalizing the feature difference values. A dynamic mode dependency matrix is ​​then constructed. The feature difference value normalization transforms the characteristic offset values ​​between modes into the average correlation index within the time series range and integrates them into the dynamic mode dependency matrix. The construction of the dependency matrix not only reflects the average interaction strength between modes, but also corrects the bias of multimodal data on the feature scale through the normalization process, so that the dependency matrix can truly reflect the relationship between modes. By iteratively updating the interaction weights, the overall interaction contribution of the corrected modal features is optimized. The modal interaction weights are updated through iterative propagation, taking into account not only their own initial weights but also the interaction strength between them and other modalities. This iterative optimization can dynamically adjust the dependencies between modalities and incorporate the impact of outlier removal into the weight update process, making the modal interaction weights more consistent with the true state of the data. By generating an optimized dynamic fusion feature tensor based on optimized modal weights, not only are multimodal features integrated, but the interactive contribution of modal weights on dynamic time series is also maximized, generating a time series feature tensor that is more suitable for global feature analysis. By dynamically fusing the feedback of the feature tensor to influence the modal weights for secondary propagation, the secondary propagation process of the modal weights is re-influenced, further optimizing the contribution of the modal to the fusion result. This design forms a closed-loop weight optimization logic, making the interaction relationship between modalities and the final weights more converge to the true state.

[0036] S3, based on the overall feature value, splits according to the mode, introduces a Gaussian kernel function to calculate the similarity of the weighted mode features of different time point pairs, and performs feature decomposition to extract low-dimensional embedding features; Preferably, based on the overall feature value, the modality is split, a Gaussian kernel function is introduced to calculate the similarity of weighted modal features of different time point pairs, and feature decomposition is performed to extract low-dimensional embedding features, including, Based on the overall feature values, the modalities are split, and a Gaussian kernel function is introduced in conjunction with the final optimized weights to calculate the similarity of the weighted modal features of different time point pairs, as shown below:

[0037] in, The scaling factor represents the Gaussian kernel function. and Representing the time after fusion and All eigenvalues, Indicates time and Weighted kernel similarity, and These represent the time intervals of the b-th mode. as well as The feature vector is composed of all the feature data, where MATLAB-style slicing, corresponding to the feature subspace of a modality; The kernel function is centered and eigenvalue decomposition is performed to extract low-dimensional embedding features, as shown below:

[0038] in, This represents the centered kernel matrix. The eigenvector matrix represents the kernel matrix decomposition. This represents a diagonal matrix, where the diagonal elements are the eigenvalues ​​of the kernel matrix. express transpose, This represents the transpose of the eigenvector matrix corresponding to the first d largest eigenvalues, where d is determined by historical data. express , is a low-dimensional representation of the embedded health features.

[0039] By calculating the weighted modal feature similarity of different time point pairs using the Gaussian kernel function, the Gaussian kernel function not only focuses on the mathematical distance between numerical values ​​in calculating the similarity between modal features, but also incorporates the influence of modal interaction weights, and dynamically presents the similarity between modal characteristics in the time series. This ensures that the modal weights have an explicit dependency on the generation of the final embedded features, rather than simply depending on the difference of the features themselves. By correcting the directional shift in the feature space through decentralized operations, in addition to the influence of the overall eigenvalues ​​in the time series on the generation of low-dimensional embedded features, eigenvalue decomposition extracts low-dimensional embedded features. This decomposes the high-dimensional data tensor from the joint influence of the number of modal features and the time series dimension, leading to a low-dimensional health feature embedding space. The eigenvalue decomposition process is not only a simple dimensionality reduction but also a way to extract the dominant factors of health features. For example, if the RR interval in the HRV modality has a core influence in dynamic health monitoring, eigenvalue decomposition will prioritize retaining the principal components of this modality while capturing the characteristic information of other interfering modalities with lower weights. The final embedded feature space generates a health-related low-dimensional representation that directly reflects the core health status after dynamic fusion of modal characteristics. This space, through the combination of weights, kernel computation, and feature decomposition, ensures that the modal characteristic dependencies can be fully expressed in the low-dimensional space, thereby making subsequent health risk assessment and prediction more accurate.

[0040] S4, calculate the health risk index, assess the health risk of different modalities at each time point, mark the risks, count the number of occurrences, and issue health monitoring alerts; Preferably, a health risk index is calculated to assess the health risk of different modalities at various time points, including, Based on the extracted low-dimensional embedding feature data, a health risk index is defined for each time point t. The health risk index is calculated by combining the low-dimensional embedding features and modality hierarchical analysis to assess the health risk of different modalities at each time point, as expressed as: ; in, The health risk index represents time t. This represents the eigenvector consisting of all feature data of the b-th mode at time t.

[0041] By constructing a health risk index through low-dimensional embedding of the feature space and modality hierarchies, the high-dimensional complexity of multimodal data is reduced, while avoiding interference from noise or redundant information in risk assessment. Combined with modality hierarchies, by assigning weights to the feature vectors of each modality, the importance of different modalities in contributing to the overall health risk can be highlighted, and these contributions can be dynamically mapped to the overall health risk assessment in the time dimension. Through the combination of global low-dimensional embedding and local refined calculation, the risk index has the characteristics of both real-time performance and global consistency, and is suitable for anomaly detection and trend prediction of health status in complex scenarios.

[0042] Furthermore, risk labeling is carried out, including, The health safety threshold is based on the sum of the mean and two standard deviations of the historical health risk index. If the calculated health risk value is greater than or equal to the health and safety threshold, then a health risk is determined to exist for modal time, and the risk is marked.

[0043] Furthermore, the frequency of occurrence will be used to generate health monitoring alerts, including: The system counts the number of times a risk marker appears within a continuous time window and sets an alarm threshold based on historical experience. If the number of triggers is greater than or equal to the alarm threshold, a health monitoring alarm is triggered.

[0044] This embodiment also provides a computer device applicable to the medical and health monitoring method based on multimodal data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the medical and health monitoring method based on multimodal data as proposed in the above embodiment.

[0045] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0046] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the medical and health monitoring method based on multimodal data as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0047] In summary, this invention achieves multidimensional expansion of modal features through observation tensor construction, extending multimodal data from time series to tensor space. This not only integrates the temporal features of single modalities but also preserves the interactivity between modalities. Through iterative updates of interaction weights, the overall interaction contribution of modal features is optimized and corrected. Modal interaction weights are updated through iterative propagation, considering not only their initial weights but also the interaction strength with other modalities. The feedback from the dynamically fused feature tensor influences the modal weights for secondary propagation, further optimizing the modal contribution to the fusion result. Finally, a health-related low-dimensional representation is generated through the embedded feature space, directly reflecting the core health status after dynamic fusion of modal characteristics.

[0048] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A medical health monitoring method based on multimodal data, characterized in that, include: Multimodal data is collected, the differences between data point pairs of different modalities are analyzed, the matching cost is defined and the cumulative cost matrix is ​​calculated, dynamic alignment of time series is performed, isolation scoring is performed and outlier removal is performed using the isolation forest algorithm, observation tensors are constructed based on the number of modalities, number of time points and modal characteristics of multimodal data, and the nuclear norm is used as a low-rank constraint term to construct the completion objective function to complete missing values; Calculate the dynamic offset between modal features and use it as the feature difference value to construct a dynamic modal dependency matrix. Combine the cross-correlation between modalities to calculate the interaction weight in the dynamic modal dependency matrix. Optimize and correct the overall interaction contribution of each modal feature to the tensor as the optimized modal weight. Then, perform weighted fusion on each time slice of the multimodal tensor to generate optimized overall feature values. Finally, perform secondary propagation to determine the final optimized weight. Based on the overall feature value, the modality is split, and a Gaussian kernel function is introduced to calculate the similarity of the weighted modal features of different time point pairs, and feature decomposition is performed to extract low-dimensional embedding features; Calculate the health risk index, assess the health risks of different modalities at different time points, mark the risks, and count the number of occurrences to generate health monitoring alerts.

2. The medical and health monitoring method based on multimodal data as described in claim 1, characterized in that: The process involves defining matching costs and calculating a cumulative cost matrix, performing dynamic time series alignment, employing the isolation forest algorithm for isolation scoring and outlier removal, constructing an observation tensor based on the number of modalities, time points, and modal features of the multimodal data, using the nuclear norm as a low-rank constraint to construct the completion objective function, and completing missing values, including... For time series data of different modalities collected, dynamic time alignment is performed. Euclidean distance is used to describe the difference between each pair of points in the modal data at different times, and it is used as the matching cost function. The cumulative cost matrix is ​​calculated using a dynamic programming algorithm, which represents the minimum cost required to align the sequence from its starting point to the desired match. Based on the final cumulative cost matrix, the aligned time trajectory is found using a backtracking method, and dynamic alignment of the time series is performed. Outlier detection is performed on the data points of each modality. The isolated forest algorithm is used to construct isolated trees and perform isolation scoring to evaluate the ease with which data points are separated in the tree. Anomaly thresholds are determined based on historical experience. If an isolated score is less than or equal to the anomaly threshold, the corresponding data point is judged to be an outlier and removed. Based on the filtered modal time series data, an observation tensor is constructed according to the number of modes, the number of time points, and modal characteristics; Calculate the Frobenius norm of the observed Zhang data, and use the nuclear norm as a low-rank constraint term to construct the completion objective function to complete the missing values; Alternating Direction Multiplier Method (ADMM) is used for target optimization, and an auxiliary tensor is introduced. This represents the low-rank approximation update objective function, and constructs the Lagrange function; The mask tensor is defined and iteratively optimized based on the Alternating Direction Multiplier Method (ADMM). The complete data tensor to be optimized is updated by element-wise multiplication based on the update objective function. Perform a soft thresholding operation on the singular value diagonal matrix, use the thresholded singular values ​​to generate a reconstructed low-rank tensor, and update the Lagrange multipliers. The complete data tensor obtained during convergence optimization is used as the supplementary data tensor.

3. The medical and health monitoring method based on multimodal data as described in claim 2, characterized in that: The optimization correction uses the modal features of each modality as the overall interaction contribution of the tensor to the optimization modal weights, and performs weighted fusion on each time slice of the multimodal tensor to generate optimized overall feature values. A second propagation is then performed to determine the final optimized weights, including... The dynamic characteristic dependencies between modes are analyzed, and the dynamic offset between modal features is calculated at each time point to quantify the instantaneous correlation between modes as the feature difference value. The feature difference values ​​are normalized and the full-time average correlation between modes is obtained. The modal dependency values ​​are analyzed to construct a dynamic modal dependency matrix. Based on the dynamic modal dependency matrix, the confidence of outliers removed by isolated scores is used as the initial interaction weight. The interaction weight of a modality is set as a combination of the weighted propagation of the interaction strength of other modalities and its own interaction weight. Iterative updates are performed. The iteration stops when the change value of the modal weight to be updated is less than or equal to the change threshold. The modal features of each modality are optimized and corrected to the overall interaction contribution of the tensor, which is used as the optimized modal weight. Based on the optimized modal weights, the multimodal tensor is weighted and fused for each time slice to generate an optimized dynamic fusion feature tensor as the overall feature value. The overall feature value is then fed back to influence the modal weights, and a second propagation is performed to further optimize the modal weights.

4. The medical and health monitoring method based on multimodal data as described in claim 3, characterized in that: The process involves splitting the overall feature value according to the modality, introducing a Gaussian kernel function to calculate the similarity of weighted modal features for pairs of different time points, and performing feature decomposition to extract low-dimensional embedding features, including... Based on the overall feature value, the modality is split, and a Gaussian kernel function is introduced in combination with the final optimized weight to calculate the similarity of the weighted modal features of different time point pairs; The kernel function is centered and eigenvalue decomposition is performed to extract low-dimensional embedding features.

5. The medical and health monitoring method based on multimodal data as described in claim 4, characterized in that: The calculation of the health risk index assesses the health risk of different modalities at various time points, including, Based on the extracted low-dimensional embedding feature data, a health risk index is defined for each time point t. The health risk index is calculated by combining the low-dimensional embedding features and modality hierarchies to assess the health risk of different modalities at each time point.

6. The medical and health monitoring method based on multimodal data as described in claim 5, characterized in that: The risk marking process includes, The health safety threshold is based on the sum of the mean and two standard deviations of the historical health risk index. If the calculated health risk value is greater than or equal to the health and safety threshold, then a health risk is determined to exist for modal time, and the risk is marked.

7. The medical and health monitoring method based on multimodal data as described in claim 6, characterized in that: The statistical frequency of occurrence is used to generate health monitoring alerts, including, The system counts the number of times a risk marker appears within a continuous time window and sets an alarm threshold based on historical experience. If the number of triggers is greater than or equal to the alarm threshold, a health monitoring alarm is triggered.

8. The medical and health monitoring method based on multimodal data as described in claim 2, characterized in that: The collection of multimodal data includes, Multimodal data acquisition is performed using physiological sensors, including heart rate data, blood oxygen saturation (SpO2) data, and heart rate variability (HRV) data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the medical and health monitoring method based on multimodal data as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the medical and health monitoring method based on multimodal data as described in any one of claims 1 to 8.

Images