Disease risk prediction method and system based on multi-modal data fusion

Abstract

The application discloses a disease risk prediction method and system based on multi-modal data fusion, relates to the technical field of intelligent medical treatment and medical data processing, and comprises the following steps: collecting multi-modal medical data of a target object, and preprocessing the multi-modal medical data to obtain standard multi-modal medical data; performing gradient zero-crossing detection to form a time sequence change segment; calculating a cross-modal minimum energy path to generate a multi-modal disease evolution feature sequence; inputting an improved DeepHit model to obtain an event time joint probability matrix; calculating cumulative occurrence probability and non-disease probability to form a disease risk evolution sequence; determining a risk grade and a risk growth interval to generate a disease risk prediction result; and generating a disease risk influence path by layer-by-layer back calculation and difference transmission. The application realizes accurate prediction of a disease risk grade, a risk growth interval and a risk influence path by combining an improved DeepHit model.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical and medical data processing technology, and in particular to a method and system for predicting disease risk based on multimodal data fusion. Background Technology

[0002] With the development of medical informatics and intelligent diagnosis and treatment technologies, electronic medical records, medical images, laboratory tests, vital signs, medication records, and lifestyle data are widely used for disease risk prediction. Existing disease risk prediction methods typically extract features from medical data from different sources and use statistical analysis or machine learning models to calculate the probability of disease occurrence. However, different modalities of medical data have different temporal variation patterns and data expression formats, making it difficult for existing methods to establish continuous and stable correlations between different medical data. This results in insufficient fusion of multimodal medical data and low accuracy in extracting disease risk features.

[0003] Existing disease risk prediction methods mainly output the probability of disease occurrence or risk level based on fusion features. They are insufficient in depicting the evolution of disease risk over time, making it difficult to accurately identify the time intervals of concentrated risk growth. Furthermore, there is a lack of clear reverse correlation between the prediction results and the original medical data, making it difficult to trace the impact path of different medical data on the formation and changes of disease risk. This results in insufficient accuracy and traceability of disease risk prediction.

[0004] Therefore, how to provide a disease risk prediction method and system based on multimodal data fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a disease risk prediction method and system based on multimodal data fusion. This invention comprehensively utilizes temporal variation extraction, cross-modal feature fusion, and deep learning risk prediction techniques, detailing the entire process from multimodal medical data acquisition and preprocessing, generation of temporal variation segments, construction of multimodal disease evolution features, calculation of joint probability of events over time, to risk level determination and tracing of impact paths. This invention improves the DeepHit model by incorporating forward and reverse channel recursive calculation modules, diagonal spectral difference folding modules, and hierarchical residual backflow modules to achieve joint prediction of disease events and their occurrence time. Compared with existing technologies, this invention has the advantages of sufficient multimodal data fusion, accurate disease risk prediction, stable prediction results, and traceable risk sources.

[0006] The disease risk prediction method based on multimodal data fusion according to embodiments of the present invention includes:

[0007] Collect multimodal medical data of the target object, perform preprocessing on the multimodal medical data, and obtain standard multimodal medical data;

[0008] The standard multimodal medical data sequence is subjected to exponential decay cumulative transformation and gradient zero-crossing detection to form time-series variation segments;

[0009] The temporal variation segments are constructed into a modal channel node graph, the cross-modal minimum energy path is calculated, an entropy uniform matrix is ​​constructed and block singular value decomposition is performed to generate a multimodal disease evolution feature sequence.

[0010] The multimodal disease evolution feature sequence is input into the improved DeepHit model to extract shared risk features, separate disease event features, and calculate the event occurrence time probability to obtain the event time joint probability matrix.

[0011] The cumulative probability of occurrence and the probability of not developing the disease are calculated based on the joint probability matrix of event time to form a disease risk evolution sequence.

[0012] By determining the risk level and risk growth range of the target disease through the disease risk evolution sequence, disease risk prediction results are generated.

[0013] Starting from the risk level and risk growth range in the disease risk prediction results, the transmission difference of the cumulative probability increment is calculated layer by layer along the multimodal disease evolution characteristic sequence to generate the disease risk impact path.

[0014] Optionally, the formation of the time-series change segment includes:

[0015] Continuous values ​​are extracted from standard multimodal medical data and arranged from earliest to latest according to the collection time. Exponential decay accumulation processing is performed on the values ​​at each time point to obtain the decay accumulation sequence corresponding to each data type.

[0016] Calculate the difference between the cumulative decay values ​​at adjacent time points in the decay cumulative sequence, divide the difference by the time interval between adjacent time points to obtain the first gradient, and determine the gradient zero-crossing position based on the positive and negative changes of the first gradient.

[0017] Gradient zero-crossing positions are arranged in chronological order, and adjacent gradient zero-crossing positions are used as segment boundaries. The cumulative decay value, gradient integral value, and duration within each segment are extracted to generate a temporally changing segment.

[0018] Optionally, the exponential decay accumulation processing on the values ​​at each time point includes:

[0019] Calculate the time difference between the current moment and each historical moment, divide the time difference by the median of the adjacent collection time intervals and take the negative value, and use the resulting value as the exponent of the natural constant to obtain the exponential decay.

[0020] Multiply the historical values ​​by the corresponding exponential decay, accumulate the products, and add the accumulated result to the current value to obtain the cumulative decay value at the current time.

[0021] The cumulative attenuation value is calculated sequentially according to the acquisition time, and the cumulative attenuation values ​​are arranged from earliest to latest to form the cumulative attenuation sequence corresponding to each data type.

[0022] Optionally, generating the multimodal disease evolution feature sequence includes:

[0023] Nodes are divided according to temporal change segments. Cross-modal connections are established for nodes of different modalities with overlapping time ranges, and sequential connections are established for adjacent nodes of the same modality before and after time, thus constructing a modal channel node graph.

[0024] Calculate the difference in cumulative attenuation, gradient integral, and duration between the two ends of each connection in the modal channel node graph. Normalize each difference and calculate the connection energy. Accumulate the connection energy layer by layer according to the node time order to obtain the cross-modal minimum energy path.

[0025] The connection energy in the cross-modal minimum energy path is converted into path weights. The path weights are normalized and entropy is calculated. The obtained values ​​are corrected for row mean, column mean and overall mean to construct an entropy uniform matrix.

[0026] The entropy uniform matrix is ​​divided into blocks, and singular value decomposition is performed on each block. The left singular vector matrix is ​​multiplied by the singular value matrix, and then the blocks are spliced ​​together according to the temporal change segments to generate a multimodal disease evolution feature sequence.

[0027] Optionally, obtaining the cross-modal minimum energy path includes:

[0028] Calculate the difference in cumulative attenuation, gradient integral, and duration between the two connected nodes, and take the absolute value of the differences.

[0029] Subtract the minimum difference of the current term from the absolute value of the difference, and then divide by the difference between the maximum and minimum differences to obtain the normalized difference. Square the normalized differences separately and add them together. Take the square root of the sum to obtain the connection energy.

[0030] The calculation is performed layer by layer according to the node time sequence. The cumulative energy of the starting node is set to 0. The cumulative energy of the previous node is added to the connection energy to the current node to obtain the candidate energy. The minimum candidate energy is selected as the cumulative energy of the current node, and the corresponding previous node is recorded.

[0031] After calculating the cumulative energy of each node in each layer, select the node with the smallest cumulative energy in the last layer, and trace back to the starting node layer by layer based on the previous node. Arrange the traced nodes in chronological order to obtain the cross-modal minimum energy path.

[0032] Optionally, obtaining the event-temporal joint probability matrix includes:

[0033] An improved DeepHit model is constructed, which includes a shared subnetwork, a disease event-specific subnetwork, and an event-temporal joint probability output layer. A forward and reverse channel recursive calculation module is set between the shared subnetwork and the disease event-specific subnetwork, and a diagonal spectral difference folding module and a hierarchical residual backflow module are set after the disease event-specific subnetwork.

[0034] The multimodal disease evolution feature sequence is input into the shared sub-network to extract shared risk features. The forward and reverse channel recursive calculation modules are used to perform forward time recursive calculation and reverse time recursive calculation respectively. The forward recursive calculation results, the reverse recursive calculation results, the difference between the two and the corresponding product are combined to obtain the disease event branch features.

[0035] The disease event branch features are input into the disease event specific sub-network. The event time initial matrix is ​​calculated according to the disease event type and future time interval. The event time initial matrix is ​​continuously modified by the diagonal spectral difference folding module to obtain the event time correction matrix.

[0036] The event time initial matrix and the event time correction matrix are added item by item and normalized. The single time interval probability, cumulative occurrence probability and non-onset probability are calculated through the hierarchical residual backflow module. The single time interval probability is corrected based on the cumulative back-calculation result to obtain the event time joint probability matrix.

[0037] Optionally, the step of performing forward and reverse time recursion through the forward and reverse channel recursion modules includes:

[0038] Read the shared risk features from early to late according to the temporal change segments, multiply the current shared risk feature by itself item by item to form an outer product matrix, and then recursively add it to the outer product matrix and feature accumulation vector of the previous time step to obtain the forward recursive calculation result;

[0039] By recursively deriving shared risk characteristics from late to early according to the temporal change segments, the reverse external accumulation matrix and the reverse feature accumulation vector are obtained, forming the reverse recursive calculation result;

[0040] Expand the forward and reverse recursive results separately, calculate the term-by-term difference and term-by-term product of the two, and then concatenate the forward and reverse recursive results, term-by-term difference and term-by-term product in sequence to obtain the disease event branch features.

[0041] Optionally, obtaining the forward recursive calculation result includes:

[0042] Set the external accumulation matrix and feature accumulation vector before the first time series change segment to 0, and read the shared risk feature vector corresponding to the current time series change segment;

[0043] Each eigenvalue in the shared risk feature vector is multiplied by all eigenvalues, and the resulting products are arranged according to their feature positions to form the current outer product matrix;

[0044] Add the current outer product matrix to the previous outer product matrix item by item, and add the current shared risk feature vector to the previous feature accumulation vector item by item to obtain the current outer product matrix and the current feature accumulation vector.

[0045] The recursive processing is repeated sequentially according to the time-series change segments until the last time-series change segment. The external accumulation matrix and feature accumulation vector at each time point are arranged in chronological order to obtain the forward recursive calculation result.

[0046] Optionally, the formation of the disease risk evolution sequence includes:

[0047] Read the joint probability matrix of event time according to the disease event type and future time interval, and determine each matrix element as the probability of the occurrence of the corresponding disease event in the corresponding future time interval;

[0048] For each disease event, the probability of occurrence in the current time interval and all previous time intervals is accumulated in the order of future time intervals to obtain the cumulative probability of occurrence in each future time interval;

[0049] Add up the cumulative probability of occurrence of each disease event within the same future time interval, and subtract the sum from 1 to obtain the probability of no disease occurrence in the corresponding future time interval;

[0050] The cumulative probability of occurrence and the probability of non-occurrence of each disease event are arranged sequentially according to future time intervals. The difference in the cumulative probability of occurrence between adjacent future time intervals is calculated to form a disease risk evolution sequence.

[0051] The disease risk prediction system based on multimodal data fusion according to an embodiment of the present invention includes the following modules:

[0052] The multimodal data processing module is used to collect multimodal medical data of the target object and perform preprocessing to obtain standard multimodal medical data;

[0053] The temporal variation extraction module is used to perform exponential decay cumulative transformation and gradient zero-crossing detection on standard multimodal medical data sequences to form temporal variation segments;

[0054] The evolutionary feature generation module is used to construct a modal channel node graph based on temporal change segments and generate a multimodal disease evolution feature sequence;

[0055] The joint probability calculation module is used to input multimodal disease evolution feature sequences into the improved DeepHit model to obtain the event-temporal joint probability matrix;

[0056] The risk evolution generation module is used to calculate the cumulative occurrence probability and non-occurrence probability of each disease event based on the event time joint probability matrix, forming a disease risk evolution sequence;

[0057] The disease risk prediction module is used to determine the risk level and risk growth range of a target disease based on the disease risk evolution sequence, and generate disease risk prediction results.

[0058] The risk path generation module is used to generate disease risk impact paths by taking the disease risk prediction results as the reverse starting point.

[0059] The beneficial effects of this invention are:

[0060] This invention improves the comprehensive utilization of multi-source medical information in disease risk prediction by integrating electronic medical records, medical images, laboratory tests, vital signs, medication records, and lifestyle data. Compared with prediction methods that rely on single data or static indicators, this invention extracts temporal variation segments of medical data through exponential decay cumulative transformation and gradient zero-crossing detection. It also combines modal channel node graphs, cross-modal minimum energy paths, entropy uniform matrices, and block singular value decomposition to fully explore the temporal correlations and evolutionary characteristics between different medical data, reducing the impact of data heterogeneity, local fluctuations, and single-modal biases on prediction results, and enhancing the integrity and stability of multimodal disease feature expression.

[0061] This invention improves the DeepHit model through forward and reverse channel recursion modules, diagonal spectral difference folding modules, and hierarchical residual backflow modules, enhancing the joint prediction capability of disease event types and occurrence times. Based on the event-time joint probability matrix, it forms a disease risk evolution sequence, enabling accurate determination of the target disease risk level and risk growth range. This invention can reverse-calculate the transmission difference of the cumulative occurrence probability increment along the multimodal disease evolution characteristic sequence to determine the source of disease risk and its evolution sequence, overcoming the problem of insufficient traceability of prediction results in existing technologies. It provides a reliable basis for early disease screening, risk classification, intervention timing determination, and personalized health management. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Figure 1 This is a flowchart of the disease risk prediction method based on multimodal data fusion proposed in this invention;

[0064] Figure 2 This is a structural block diagram of the improved DeepHit model, which is based on multimodal data fusion for disease risk prediction proposed in this invention.

[0065] Figure 3 This is a functional diagram of the disease risk prediction system based on multimodal data fusion proposed in this invention. Detailed Implementation

[0066] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0067] refer to Figure 1 and Figure 2 Disease risk prediction methods based on multimodal data fusion include:

[0068] Collect multimodal medical data of the target object, perform preprocessing on the multimodal medical data, and obtain standard multimodal medical data;

[0069] The standard multimodal medical data sequence is subjected to exponential decay cumulative transformation and gradient zero-crossing detection to form time-series variation segments;

[0070] The temporal variation segments are constructed into a modal channel node graph, the cross-modal minimum energy path is calculated, an entropy uniform matrix is ​​constructed and block singular value decomposition is performed to generate a multimodal disease evolution feature sequence.

[0071] The multimodal disease evolution feature sequence is input into the improved DeepHit model to extract shared risk features, separate disease event features, and calculate the event occurrence time probability to obtain the event time joint probability matrix.

[0072] The cumulative probability of occurrence and the probability of non-occurrence of each disease event are calculated based on the joint probability matrix of event time, forming a disease risk evolution sequence;

[0073] By determining the risk level and risk growth range of the target disease through the disease risk evolution sequence, disease risk prediction results are generated.

[0074] Starting from the risk level and risk growth range in the disease risk prediction results, the transmission difference of the cumulative probability increment is calculated layer by layer along the multimodal disease evolution characteristic sequence to generate the disease risk impact path.

[0075] Multimodal medical data includes electronic medical record data, medical imaging data, laboratory test data, vital sign data, medication record data, and lifestyle behavior data.

[0076] Standardized multimodal medical data were obtained, specifically:

[0077] Multimodal medical data is associated with the identity of the target object, and the collection time, data source and data type are recorded and arranged by collection time to form the original medical data sequence;

[0078] Unify the time reference of the original medical data sequence, delete duplicate records, mark data that are outside the effective range of medical indicators, and fill in continuous missing data according to the effective values ​​and time intervals before and after;

[0079] The processed medical data undergoes medical terminology coding, unit conversion, image resolution standardization, and numerical normalization to obtain standard multimodal medical data. Specifically, the obtained standard multimodal medical data includes:

[0080] Electronic medical records, test results, drug names, and lifestyle descriptions are matched with a medical terminology coding table. When the name similarity reaches 0.90, the highest similarity code is assigned; when it is below 0.90, a verification pending label is set. The medical images are scaled while maintaining their aspect ratio and zero-padding is applied to 512×512 pixels. Out-of-bounds values ​​are corrected according to the effective lower and upper limits of each medical indicator. The current value is subtracted from the effective lower limit, and the difference is divided by the difference between the effective upper and lower limits to normalize the value to the range of 0 to 1. The data is then arranged according to the target object, data type, and collection time to obtain standard multimodal medical data.

[0081] In this embodiment, the formation of the time-series change segment includes:

[0082] Continuous values ​​are extracted from standard multimodal medical data and arranged from earliest to latest according to the collection time. Exponential decay accumulation processing is performed on the values ​​at each time point to obtain the decay accumulation sequence corresponding to each data type.

[0083] Calculate the difference between the cumulative decay values ​​at adjacent time points in the decay cumulative sequence, divide the difference by the time interval between adjacent time points to obtain the first gradient, and determine the gradient zero-crossing position based on the positive and negative changes of the first gradient.

[0084] Gradient zero-crossing positions are arranged in chronological order, and adjacent gradient zero-crossing positions are used as segment boundaries. The cumulative decay value, gradient integral value, and duration within each segment are extracted to generate a temporally changing segment.

[0085] In this embodiment, exponential decay accumulation processing is performed on the values ​​at each time point, including:

[0086] Calculate the time difference between the current moment and each historical moment, divide the time difference by the median of the adjacent collection time intervals and take the negative value, and use the resulting value as the exponent of the natural constant to obtain the exponential decay.

[0087] Multiply the historical values ​​by the corresponding exponential decay, accumulate the products, and add the accumulated result to the current value to obtain the cumulative decay value at the current time.

[0088] The cumulative attenuation value is calculated sequentially according to the acquisition time, and the cumulative attenuation values ​​are arranged from earliest to latest to form the cumulative attenuation sequence corresponding to each data type.

[0089] In this embodiment, generating multimodal disease evolution feature sequences includes:

[0090] Nodes are divided according to temporal change segments. Cross-modal connections are established for nodes of different modalities with overlapping time ranges, and sequential connections are established for adjacent nodes of the same modality before and after a certain time. A modal channel node graph is constructed, which is specifically formed as follows:

[0091] Read each time series change segment according to the data mode. Combine the mode type, start time, end time, cumulative decay value, gradient integral value and duration of each segment into 1 node. For different mode nodes, take the smaller value of the end time of the two segments and subtract the larger value of the start time. If the result is greater than 0, it is determined that there is an overlap in the time range. Establish a cross-modal connection between the two nodes. For the same mode node, arrange the segments in order of the start time from the earliest to the latest and establish directed connections between adjacent segments in the order of the last segment. Subtract the end time of the previous segment from the start time of the next segment to obtain the transition interval. Record the segment characteristics of each node, the type, direction, overlap duration and transition interval of each connection, and form a modal channel node diagram.

[0092] Calculate the cumulative attenuation difference, gradient integral difference, and duration difference between the two ends of each connection in the modal channel node graph. Normalize each difference and calculate the connection energy. Accumulate the connection energy layer by layer according to the node time sequence to obtain the minimum energy path across modes. Specifically, the calculation of the cumulative attenuation difference, gradient integral difference, and duration difference between the two ends of each connection in the modal channel node graph is as follows:

[0093] Read the temporal change segments corresponding to the two ends of each connection in the modal channel node graph, and extract the cumulative decay value, gradient integral value, and segment start and end time of the two ends nodes respectively. Subtract the cumulative decay value of the first node from the cumulative decay value of the second node and take the absolute value to obtain the cumulative decay value difference. Subtract the gradient integral value of the first node from the gradient integral value of the second node and take the absolute value to obtain the gradient integral value difference. Subtract the segment start time from the segment end time of each node to obtain the duration. Subtract the durations of the two nodes and take the absolute value to obtain the duration difference. Use the three differences as the node difference data of the corresponding connection.

[0094] The connection energies in the cross-modal minimum energy path are converted into path weights. These path weights are then normalized and entropy is calculated. The resulting values ​​are corrected for row mean, column mean, and overall mean to construct an entropy uniformity matrix. Specifically, the construction of the entropy uniformity matrix is ​​as follows:

[0095] The energy of each connection in the cross-modal minimum energy path is incremented by 1 and the reciprocal is taken to obtain the connection transformation value. The connection transformation value is normalized according to the sum of the connection transformation values ​​within the same temporal change segment to obtain the path probability of each connection, so that the sum of the path probabilities corresponding to the same temporal change segment is 1. The negative value of the product of each path probability and the natural logarithm is calculated, and the result is used as the information entropy contribution value of the connection to the temporal change segment. All information entropy contribution values ​​are arranged with the temporal change segment as the row and the modality number as the column to form the information entropy contribution moment. Each matrix element represents the contribution of a single connection to the overall information entropy of the row probability distribution. The complete information entropy corresponding to the temporal change segment is obtained by summing the elements of each row.

[0096] The average value of the information entropy contribution value of each row, the average value of the information entropy contribution value of each column, and the overall average value of all matrix elements are calculated separately. For each information entropy contribution value, the mean value of the row is subtracted first, then the mean value of the column is subtracted, and finally the overall mean value of the matrix is ​​added to obtain the corrected information entropy contribution value. An entropy uniform matrix is ​​formed according to the original row and column positions. The overall offset caused by different time segments and different modal connections is corrected bidirectionally, so that the matrix retains the local information difference of each connection relative to its time segment and modal channel, while reducing the common offset in the row and column directions.

[0097] The entropy uniform matrix is ​​divided into blocks, and singular value decomposition is performed on each block. The left singular vector matrix is ​​multiplied by the singular value matrix, and the blocks are then concatenated according to temporal variation segments to generate a multimodal disease evolution feature sequence. Specifically, the entropy uniform matrix is ​​divided into blocks, and singular value decomposition is performed on each block.

[0098] Based on the column indices of electronic medical records, medical images, laboratory tests, vital signs, medication records, and lifestyle data in the entropy uniform matrix, the starting and ending columns of each data modality are determined. While keeping the corresponding rows of all time-series change segments unchanged, each modality matrix is ​​segmented. The product of each matrix segment and the transpose matrix is ​​calculated. The product results are decomposed into eigenvalues, and the eigenvalues ​​are arranged in descending order. The square root of each eigenvalue is taken to obtain the singular values. The corresponding eigenvectors are arranged in the same order to form a left singular vector matrix. The left singular vector matrix is ​​multiplied by the singular value matrix to obtain the compressed feature results of each time-series change segment under the data modality. The compressed feature results of each matrix segment are then concatenated row by row according to the modality order.

[0099] In this embodiment, obtaining the cross-modal minimum energy path includes:

[0100] Calculate the difference in cumulative attenuation, gradient integral, and duration between the two connected nodes, and take the absolute value of the differences.

[0101] Subtracting the minimum difference from the absolute value of the difference, and then dividing by the difference between the maximum and minimum differences, yields the normalized difference. Squaring each of these normalized differences and summing them, then taking the square root of the sum, gives the connection energy. Specifically, the connection energy is obtained as follows:

[0102]

[0103]

[0104]

[0105]

[0106] in, This represents the connection energy between node i and node j. This represents the cumulative difference in attenuation between the two nodes. This represents the gradient integral difference between the two endpoints. This represents the difference in duration between the two nodes. and These represent the minimum and maximum values ​​of the cumulative attenuation difference across all connections, respectively. and These represent the minimum and maximum values ​​of the gradient integral difference, respectively. and These represent the minimum and maximum values ​​of the duration difference, respectively. This represents the normalized value of the cumulative difference in attenuation. This represents the normalized value of the gradient integral difference. The normalized value representing the duration difference;

[0107] The calculation is performed layer by layer according to the node time sequence. The cumulative energy of the starting node is set to 0. The cumulative energy of the previous node is added to the connection energy to the current node to obtain the candidate energy. The minimum candidate energy is selected as the cumulative energy of the current node, and the corresponding previous node is recorded.

[0108] After calculating the cumulative energy of each node in each layer, select the node with the smallest cumulative energy in the last layer, and trace back to the starting node layer by layer based on the previous node. Arrange the traced nodes in chronological order to obtain the cross-modal minimum energy path.

[0109] In this embodiment, obtaining the joint probability matrix of event times includes:

[0110] An improved DeepHit model is constructed, comprising a shared subnetwork, a disease event-specific subnetwork, and a joint probability output layer for event time. A forward and inverse channel recursive module is placed between the shared subnetwork and the disease event-specific subnetwork. A diagonal spectral difference folding module and a hierarchical residual backflow module are placed after the disease event-specific subnetwork. Specifically, the improved DeepHit model is constructed as follows:

[0111] This invention structurally optimizes the traditional DeepHit model to construct an improved DeepHit model suitable for multimodal disease risk prediction. It retains the shared subnetwork, disease event-specific subnetwork, and event time joint probability output layer. The forward computation path is established in the order of multimodal disease evolution feature input, shared risk extraction, disease event branch processing, and joint probability output, thus maintaining the original model's ability to jointly model competing disease events and their occurrence time.

[0112] A forward and reverse channel recursive calculation module is set up between the shared subnetwork and the disease event-specific subnetwork. The risk features output by the shared subnetwork are accumulated layer by layer in forward and reverse time order, respectively. The bidirectional change information of disease risk from early state to backward and from current state to forward is extracted. The forward results, reverse results and their differences are then combined and input into the disease event-specific subnetwork, which effectively enhances the model's ability to identify the disease evolution sequence, the dependency relationship between the past and the future and the long-term risk changes.

[0113] The diagonal spectral difference folding module extracts diagonal sequences from the initial event time matrix according to the direction of synchronous change in the disease event dimension and the prediction time dimension. This ensures that each sequence simultaneously contains information on the differences between disease events and the probability changes over continuous time intervals. The module performs spectral transformation on each diagonal sequence, calculates the first-order difference between adjacent spectral components and the second-order difference between the first-order differences, determines the location of local probability mutations based on the second-order differences, and corrects the mutation spectral components. The module then performs inverse spectral transformation on the corrected spectral sequence and folds it according to the original row and column positions of each element in the initial event time matrix to obtain the event time correction matrix. This allows the probability correction to simultaneously combine changes in disease events with continuous changes over time.

[0114] A hierarchical margin backflow module is set up between the event time joint probability output layer and the error backflow path. The hierarchical margin is calculated based on the single time interval probability, cumulative occurrence probability and non-occurrence probability of each disease event. The difference between the cumulative result and the single interval result is back-propagated along the model hierarchy as an additional error to participate in the model parameter update, so that the prediction results of each time interval maintain a progressive relationship and reduce the problem of inconsistency between the cumulative probability and the single interval probability.

[0115] The improved DeepHit model retains the ability of traditional models to jointly predict multiple disease events and their occurrence times, while also realizing bidirectional time series information recursion, continuous probability change correction, and hierarchical residual return. It can make fuller use of the multimodal disease evolution characteristics, improve the accuracy, stability, and temporal continuity of disease risk prediction, and provide a reliable foundation for risk level determination, risk growth interval identification, and disease risk impact path tracing.

[0116] The multimodal disease evolution feature sequence is input into the shared sub-network to extract shared risk features. The forward and reverse channel recursive calculation modules are used to perform forward time recursive calculation and reverse time recursive calculation respectively. The forward recursive calculation results, the reverse recursive calculation results, the difference between the two and the corresponding product are combined to obtain the disease event branch features.

[0117] The disease event branch features are input into the disease event-specific sub-network. An initial event time matrix is ​​calculated according to the disease event type and future time interval. The initial event time matrix is ​​then continuously modified using a diagonal spectral difference folding module to obtain the modified event time matrix, where:

[0118] The disease event branch features are input into the disease event specific sub-network. Specifically, the disease event branch features are input into each disease event processing branch. The feature template formed during the training process is called. The absolute difference between the input feature value and the template feature value is calculated dimension by dimension. The absolute difference is divided by the difference between the upper limit and the lower limit of the current dimension feature value to obtain the normalized difference. The normalized difference is subtracted from 1 to obtain the matching weight. The input feature value of each dimension is multiplied by the corresponding matching weight. All weighted results are summed according to the time sequence change segments to obtain the disease event response value of each segment. The response value of each segment is multiplied by the time interval weight obtained during training and the correction value is accumulated to form the initial output of each future time interval. The initial event time matrix is ​​obtained by arranging the data row by row and column by column according to the disease event type.

[0119] The event time initial matrix is ​​continuously modified by the diagonal spectral difference folding module. Specifically, the diagonal extraction direction is taken from the upper left corner to the lower right corner of the event time initial matrix. First, the elements on the main diagonal are extracted. Then, the parallel diagonals above and below the main diagonal are extracted in sequence. For each diagonal, the elements are read in the order of synchronously increasing disease event row number and time interval column number. The row and column positions of each element in the original matrix are recorded. For diagonals whose length is less than the longest diagonal sequence, 0 is padded at the end of the sequence to make all diagonal sequences have the same length. The diagonals are arranged in ascending order of distance from the main diagonal to obtain the diagonal sequence used for spectral difference calculation.

[0120] The event time initial matrix and the event time correction matrix are added item by item and normalized. The single time interval probability, cumulative occurrence probability and non-onset probability are calculated through the hierarchical residual backflow module. The single time interval probability is corrected based on the cumulative back-calculation result to obtain the event time joint probability matrix.

[0121] In this embodiment, the step of performing forward time recursion and reverse time recursion through the forward and reverse channel recursion modules respectively includes:

[0122] Read the shared risk features from early to late according to the temporal change segments, multiply the current shared risk feature by itself item by item to form an outer product matrix, and then recursively add it to the outer product matrix and feature accumulation vector of the previous time step to obtain the forward recursive calculation result;

[0123] By recursively deriving shared risk characteristics from late to early according to the temporal change segments, the reverse external accumulation matrix and the reverse feature accumulation vector are obtained, forming the reverse recursive calculation result;

[0124] Expand the forward and reverse recursive results separately, calculate the term-by-term difference and term-by-term product of the two, and then concatenate the forward and reverse recursive results, term-by-term difference and term-by-term product in sequence to obtain the disease event branch features.

[0125] In this embodiment, obtaining the forward recursive calculation result includes:

[0126] Set the external accumulation matrix and feature accumulation vector before the first time series change segment to 0, and read the shared risk feature vector corresponding to the current time series change segment;

[0127] Each eigenvalue in the shared risk feature vector is multiplied by all eigenvalues, and the resulting products are arranged according to their feature positions to form the current outer product matrix;

[0128] Add the current outer product matrix to the previous outer product matrix item by item, and add the current shared risk feature vector to the previous feature accumulation vector item by item to obtain the current outer product matrix and the current feature accumulation vector.

[0129] The recursive processing is repeated sequentially according to the time-series change segments until the last time-series change segment. The external accumulation matrix and feature accumulation vector at each time point are arranged in chronological order to obtain the forward recursive calculation result.

[0130] In this embodiment, forming a disease risk evolution sequence includes:

[0131] Read the joint probability matrix of event time according to the disease event type and future time interval, and determine each matrix element as the probability of the occurrence of the corresponding disease event in the corresponding future time interval;

[0132] For each disease event, the probability of occurrence in the current time interval and all previous time intervals is accumulated in the order of future time intervals to obtain the cumulative probability of occurrence in each future time interval;

[0133] Add up the cumulative probability of occurrence of each disease event within the same future time interval, and subtract the sum from 1 to obtain the probability of no disease occurrence in the corresponding future time interval;

[0134] The cumulative probability of occurrence and the probability of non-occurrence of each disease event are arranged sequentially according to future time intervals. The difference in the cumulative probability of occurrence between adjacent future time intervals is calculated to form a disease risk evolution sequence.

[0135] The generation of disease risk prediction results includes:

[0136] Extract the cumulative occurrence probability and probability increment of the target disease in each future time interval from the disease risk evolution sequence, and arrange them in chronological order;

[0137] Based on the probability increments and the degree of continuous change in each future time interval, identify continuous time intervals where the cumulative probability of occurrence continues to rise and the growth rate is concentrated, thus determining the risk growth interval. Specifically, determining the risk growth interval involves:

[0138] Read the probability increments of the target disease in future time sequence, calculate the average of the probability increments of all future time intervals, and mark the time intervals with probability increments greater than 0 and not lower than the average as growth intervals. For adjacent growth intervals, divide the absolute value of the difference between their probability increments by the larger of the two values. If the result is not greater than 0.50, the growth change is considered continuous. Merge at least two consecutive time intervals into candidate intervals, accumulate the probability increments in each candidate interval, and then divide by the sum of the probability increments of all future time intervals. If the percentage of the result reaches 0.60, the growth amplitude is considered concentrated. Select the candidate interval with the largest percentage of probability increments, and determine the start and end times as risk growth intervals.

[0139] Based on the cumulative incidence probability, probability increment, and non-occurrence probability of the target disease, combined with the risk grading range formed by the training samples, the risk level of the target disease is determined. Specifically, determining the risk level of the target disease involves:

[0140] From the target disease risk growth range, extract the cumulative occurrence probability, maximum probability increment, and minimum probability of no onset. Multiply the cumulative occurrence probability by 0.50, the maximum probability increment by 0.30, and the minimum probability of no onset by 0.20. Add the three results to obtain the comprehensive risk value of the target disease. Use the same calculation method to obtain the comprehensive risk value of the training samples. Sort them in ascending order and divide them into 3 groups. Use the maximum value of the first group and the maximum value of the second group as two classification boundaries. If the comprehensive risk value of the target disease is not higher than the first boundary, it is determined to be low risk. If it is higher than the first boundary but not higher than the second boundary, it is determined to be medium risk. If it is higher than the second boundary, it is determined to be high risk.

[0141] It gathers the risk level, risk growth range, and corresponding cumulative probability changes of the target disease to generate disease risk prediction results.

[0142] The generated disease risk impact pathway includes:

[0143] Extract the target disease risk level and risk growth interval from the disease risk prediction results, read the cumulative probability increment of occurrence for each future time interval within the risk growth interval, and set the reverse calculation starting point in reverse time order. Specifically, setting the reverse calculation starting point in reverse time order is as follows:

[0144] Extract the start time, end time, and cumulative probability increment of each future time interval within the risk growth interval. Rearrange each time interval in the direction from the end time to the start time. Set the time interval at the end of the risk growth interval as the reverse sequence number 1. Read the cumulative probability increment of the time interval as the initial back calculation value. If the increment is 0, search backwards until the nearest time interval with an increment greater than 0 is found and set as the reverse sequence number 1. Use the current reverse sequence number to increment by 1 to mark the previous time interval. At the same time, record the cumulative probability, probability increment, and target disease risk level of each time interval to form the reverse back calculation starting point and back calculation order that traces back layer by layer from the end of the risk growth interval to the beginning.

[0145] Each cumulative probability increment is sequentially fed back to the event-time joint probability matrix and the multimodal disease evolution feature sequence. The risk response difference before and after the change of input features at each layer is calculated to obtain the incremental transfer value at each evolution feature position. Specifically, the risk response difference before and after the change of input features at each layer is calculated as follows:

[0146] For each future time interval within the risk growth interval, the original cumulative probability increment of the target disease in the time interval is first used as the baseline risk response. Following the reverse order of the event time joint probability matrix to the multimodal disease evolution feature sequence, one input feature position is selected layer by layer. The feature value of the current position is replaced with the feature value of the previous time position, the first time position is replaced with 0, and the remaining inputs remain unchanged. The cumulative probability increment of the target disease is recalculated using the original calculation path after this layer. The risk response difference of the feature position is obtained by subtracting the recalculated probability increment from the baseline risk response.

[0147] Take the absolute value of the risk response difference for all feature locations in the same layer and sum them up. Divide each risk response difference by the sum and multiply it by the current cumulative probability increment of the backhaul. The result with positive and negative directions is used as the incremental transmission value of the evolution feature location.

[0148] Based on the correspondence between multimodal disease evolution feature sequences and modal channel nodes and temporal change segments, incremental transfer values ​​are regressed to each medical data modality to identify risk sources. Specifically, identifying risk sources involves:

[0149] Based on the feature location index, the incremental transfer values ​​corresponding to the same medical data modality and the same temporal change segment are added together to obtain the segment transfer value of each level. For adjacent levels, the segment transfer value of the next level is subtracted from the segment transfer value of the previous level to obtain the level transfer difference. A positive difference indicates that the current level enhances risk transfer, and a negative difference indicates that the current level weakens risk transfer. For adjacent temporal change segments within the same level, the segment transfer value of the next level is subtracted from the segment transfer value of the previous level to obtain the time transfer difference. The absolute value of the level transfer difference corresponding to each medical data modality and segment is added to the absolute value of the time transfer difference to obtain the source strength. The average value of all source strengths is calculated. Medical data modalities and temporal change segments with positive incremental transfer values ​​and source strengths not lower than the average value are identified as risk sources and arranged in descending order of source strength.

[0150] Each risk source is arranged according to its corresponding medical data modality, temporal variation segment, and direction of transmission difference. Risk sources with continuous transmission relationships are connected to generate disease risk impact paths. Specifically, the generation of disease risk impact paths involves:

[0151] Each risk source is combined with its corresponding medical data modality, start and end time of the temporal change segment, hierarchical number, direction of transmission difference, and source strength into a risk node. The nodes are arranged from earliest to latest segment time and from input layer to output layer hierarchical number. When two risk nodes are connected in the modal channel node graph, have adjacent temporal change segments or overlapping time ranges, have hierarchical numbers that differ by 1, and the product of the two transmission differences is greater than 0, they are determined to have a continuous transmission relationship. A directed connection is established from the previous node to the next node. The smaller of the absolute values ​​of the transmission differences at both ends of the connection is taken as the connection strength. The source strength and connection strength of each risk node in each continuous path are accumulated to obtain the path influence value. The continuous path with the largest path influence value is selected and output sequentially according to the medical data modality, temporal change segment, and direction of risk enhancement or risk weakening to generate the disease risk influence path.

[0152] refer to Figure 3 The disease risk prediction system based on multimodal data fusion includes the following modules:

[0153] The multimodal data processing module is used to collect multimodal medical data of the target object and perform preprocessing to obtain standard multimodal medical data;

[0154] The temporal variation extraction module is used to perform exponential decay cumulative transformation and gradient zero-crossing detection on standard multimodal medical data sequences to form temporal variation segments;

[0155] The evolutionary feature generation module is used to construct a modal channel node graph based on temporal change segments and generate a multimodal disease evolution feature sequence;

[0156] The joint probability calculation module is used to input multimodal disease evolution feature sequences into the improved DeepHit model to obtain the event-temporal joint probability matrix;

[0157] The risk evolution generation module is used to calculate the cumulative occurrence probability and non-occurrence probability of each disease event based on the event time joint probability matrix, forming a disease risk evolution sequence;

[0158] The disease risk prediction module is used to determine the risk level and risk growth range of a target disease based on the disease risk evolution sequence, and generate disease risk prediction results.

[0159] The risk path generation module is used to generate disease risk impact paths by taking the disease risk prediction results as the reverse starting point.

[0160] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-disease joint follow-up scenario in a comprehensive medical institution. The method of this invention was used to predict the risk of ischemic cardiac events, with cerebrovascular events as competing disease events. 1800 samples with complete multimodal follow-up information were selected, of which 420 had the target disease event, 280 had competing disease events, and 1100 did not have the aforementioned disease event at the observation cutoff and were marked as censored samples. 1200 of these samples were used as training samples, 300 as validation samples, and 300 as test samples. The traditional DeepHit method and this invention used the exact same data partitioning.

[0161] The 1800 samples contained 19440 electronic medical records, 12960 medical images, 26280 laboratory test records, 40320 vital sign records, 16380 medication records, and 29700 daily life activity records. In the raw data, duplicate records accounted for 2.6%, consecutive missing values ​​for 4.7%, and records with inconsistent units of measurement accounted for 6.1%. During processing, duplicate records were removed, consecutive missing values ​​were filled in based on consecutive valid values ​​and time intervals, the units of measurement for medical indicators were standardized, medical images were adjusted to 512×512 pixels, and consecutive values ​​were normalized to the 0-1 range.

[0162] Taking a sample of a target disease event in the test set as an example, this sample contains 12 electronic medical records, 8 medical images, 16 laboratory test records, 24 vital sign records, 10 medication records, and 18 lifestyle behavior records. After processing, the normalized value of systolic blood pressure increased from 0.42 to 0.71, LDL cholesterol increased from 0.38 to 0.69, fasting blood glucose increased from 0.35 to 0.54, carotid intima-media thickness increased from 0.46 to 0.73, medication adherence rate decreased from 0.91 to 0.67, and activity level decreased from 0.76 to 0.39.

[0163] Continuous numerical values ​​were extracted according to data type and exponential decay accumulation processing was performed. Taking systolic blood pressure as an example, the cumulative decay values ​​at six consecutive observation positions were 0.42, 0.58, 0.79, 1.06, 1.41, and 1.83, respectively, with first-order gradients at adjacent positions of 0.16, 0.21, 0.27, 0.35, and 0.42, respectively. Based on the positive and negative changes in the first-order gradients in each medical data modality, segments were divided, resulting in 14 time-series change segments, including segments of decreased activity levels, decreased medication adherence, persistently elevated blood lipids, persistently elevated blood pressure, and changes in imaging structure.

[0164] Fourteen temporal variation segments were designated as nodes. Cross-modal connections were established for nodes with overlapping time ranges but different modalities, and sequential connections were established for adjacent nodes of the same modality, resulting in a total of 27 connections. Taking the connection between the blood lipid segment and the blood pressure segment as an example, the difference in their cumulative decay value was 0.31, the difference in their gradient integral value was 0.24, and the difference in their duration was 0.18. After normalization, the connection energy was calculated to be 0.43. The connection energy was accumulated layer by layer according to the temporal order of the nodes. The minimum cumulative energy was 1.84, and the second minimum cumulative energy was 2.31. The minimum energy path across modalities was obtained by tracing back.

[0165] To construct the entropy uniform matrix, each temporal variation segment is set as one row. Six medical data modalities—electronic medical records, medical imaging, laboratory tests, vital signs, medication records, and lifestyle behaviors—are each assigned three fixed feature positions: cumulative decay difference, gradient integral difference, and duration difference, forming 18 columns. The normalized differences of the 27 connections are multiplied by the path weights and accumulated to the corresponding feature positions according to the medical data modality of the other end node, forming a 14-row, 18-column connection contribution matrix. Each row is normalized, and the negative of the product of each normalization result and its natural logarithm is calculated to form the information entropy contribution matrix. Finally, the mean of the row and column are subtracted, and the overall matrix mean is added to obtain the 14-row, 18-column entropy uniform matrix.

[0166] Based on the column intervals corresponding to the six medical data modalities, the 14x18 entropy uniform matrix was divided into six 14x3 matrix blocks. Singular value decomposition was performed on each matrix block, retaining the first two principal singular features for each block. The cumulative singular value contribution rate of the retained features all exceeded 0.90, resulting in six 14x2 compressed feature results. While maintaining the row positions of the 14 time-series variation segments, the six compressed feature results were concatenated column-wise according to the order of the medical data modalities, forming a 14x12 multimodal disease evolution feature sequence.

[0167] A 14-row, 12-column multimodal disease evolution feature sequence is input into the improved DeepHit model, and a shared sub-network yields a 14-row, 32-column shared risk feature. The forward and reverse recursive calculation modules calculate the external accumulation matrix and feature accumulation vector in forward and reverse chronological order, respectively. The average forward recursive result is 0.47, the average reverse recursive result is 0.39, and the average difference between the two is 0.08. The forward and reverse recursive results, their differences, and their product are combined and input into the disease event-specific sub-network to obtain the initial event time matrix for the target disease and competing diseases in six future time intervals.

[0168] For each disease event's continuous-time output, its probability sequence is shifted forward by one time interval, kept in place, and shifted backward by one time interval to form a three-layer local time matrix. A local diagonal sequence of length 3 is extracted along the synchronous change direction of the layer position and time position, with the boundary positions extended using the nearest effective value. A spectral transform is performed on each local diagonal sequence, calculating the first-order difference between adjacent spectral components and the second-order difference between adjacent first-order differences. A probability correction is formed based on the second-order difference, and then the original time position is restored through inverse spectral transform. The corrections for the six time intervals of the target disease are 0.006, -0.002, 0.005, 0.014, 0.018, and 0.004, respectively. After joint normalization, these corrections are processed using hierarchical residual backflow.

[0169] After hierarchical residual backflow processing, the single-time-interval probabilities of the target disease in the six future time intervals are 0.052, 0.058, 0.101, 0.112, 0.126, and 0.110, respectively, with cumulative occurrence probabilities of 0.052, 0.110, 0.211, 0.323, 0.449, and 0.559, respectively; the cumulative occurrence probabilities of competing disease events are 0.028, 0.045, 0.061, 0.079, 0.096, and 0.108, respectively. Subtracting the sum of the cumulative occurrence probabilities of the two disease events in the same time interval from 1 yields the non-disease probabilities of 0.920, 0.845, 0.728, 0.598, 0.455, and 0.333, respectively. The average difference between the cumulative back-calculated probability and the single-time-interval probability decreased from 0.026 before backflow to 0.004.

[0170] During training, the log-likelihood error of this invention decreased from 0.934 to 0.286, the risk ranking error decreased from 0.412 to 0.097, the hierarchical residual error decreased from 0.138 to 0.021, and the validation set joint error decreased from 1.484 to 0.421. Training ended after the joint error remained below 0.001 for eight consecutive rounds. Under the same conditions of 1200 training samples and 300 validation samples, the validation set joint error of the traditional DeepHit method was 0.563.

[0171] For the aforementioned test samples, the average probability increment of the target disease across the six time intervals is 0.559 divided by 6, resulting in 0.093. The probability increments for time intervals 3 to 6 are 0.101, 0.112, 0.126, and 0.110, respectively, all of which are not lower than the average value, and the percentage change between adjacent probability increments is no greater than 0.50. Therefore, time intervals 3 to 6 are merged into a candidate growth interval. The sum of the probability increments across the four time intervals is 0.449, accounting for 80.3% of the total probability increment of 0.559, which is higher than 0.60. Thus, time intervals 3 to 6 are determined as risk growth intervals. Multiplying the cumulative probability of occurrence at the end of the interval (0.559) by 0.50, the maximum probability increment (0.126) by 0.30, and the result of subtracting the minimum probability of no disease (0.333) from 1 by 0.20, the sum of these three factors yields a comprehensive risk value of 0.451, which is higher than the high-risk boundary of 0.397 formed by the training samples. Therefore, it is determined to be high-risk.

[0172] Backward calculation from the end of the risk growth interval revealed source intensities of 0.294, 0.261, 0.224, 0.143, 0.118, and 0.076 for medical imaging, laboratory tests, vital signs, medication records, lifestyle behaviors, and electronic medical records, respectively, with an average source intensity of 0.186. Medical imaging, laboratory tests, and vital signs, with source intensities not lower than the average and positive incremental transfer values, were identified as primary risk sources. Medication records and lifestyle behaviors, although lower than the average, had positive incremental transfer values ​​and were continuously connected to the primary risk sources in the same direction, thus identified as associated risk sources. Electronic medical records did not form a continuous, unidirectional transfer relationship and were not included in the final path. Connecting the temporal change segments and transfer directions yielded a disease risk impact path: decreased activity level – decreased medication adherence – increased blood lipids – persistently elevated blood pressure – carotid artery structural abnormalities – increased disease risk.

[0173] On the same 300 test samples, the present invention was compared with the traditional DeepHit method. The time consistency index of the traditional DeepHit method was 0.761, while that of the present invention was 0.842; the area under the time dependency curve of the traditional DeepHit method was 0.782, while that of the present invention was 0.869; the comprehensive Brillouin error of the traditional DeepHit method was 0.168, while that of the present invention was 0.112; the recall rate of high-risk samples of the traditional DeepHit method was 72.6%, while that of the present invention was 88.4%; the false positive rate of the traditional DeepHit method was 15.8%, while that of the present invention was 7.9%; the accuracy rate of event occurrence time interval identification of the traditional DeepHit method was 66.7%, while that of the present invention was 81.9%. In test samples with clear risk source labels, the main risk source matching rate of the traditional DeepHit method was 58.9%, while that of the present invention was 84.4%.

[0174] As can be seen from this embodiment, the present invention can improve the accuracy of predicting disease events and their occurrence time, and identify risk growth intervals and disease risk impact paths through intermediate computational data, thus solving the problems of insufficient multimodal correlation, delayed risk discovery, and lack of source explanation for prediction results in traditional methods.

[0175] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A disease risk prediction method based on multimodal data fusion, characterized in that, include: Collect multimodal medical data of the target object, perform preprocessing on the multimodal medical data, and obtain standard multimodal medical data; The standard multimodal medical data sequence is subjected to exponential decay cumulative transformation and gradient zero-crossing detection to form time-series variation segments; The temporal variation segments are constructed into a modal channel node graph, the cross-modal minimum energy path is calculated, an entropy uniform matrix is ​​constructed and block singular value decomposition is performed to generate a multimodal disease evolution feature sequence. The multimodal disease evolution feature sequence is input into the improved DeepHit model to extract shared risk features, separate disease event features, and calculate the event occurrence time probability to obtain the event time joint probability matrix. The cumulative probability of occurrence and the probability of not developing the disease are calculated based on the joint probability matrix of event time to form a disease risk evolution sequence. By determining the risk level and risk growth range of the target disease through the disease risk evolution sequence, disease risk prediction results are generated. Starting from the risk level and risk growth range in the disease risk prediction results, the transmission difference of the cumulative probability increment is calculated layer by layer along the multimodal disease evolution characteristic sequence to generate the disease risk impact path.

2. The disease risk prediction method based on multimodal data fusion according to claim 1, characterized in that, The formation of the time-series change segment includes: Continuous values ​​are extracted from standard multimodal medical data and arranged from earliest to latest according to the collection time. Exponential decay accumulation processing is performed on the values ​​at each time point to obtain the decay accumulation sequence corresponding to each data type. Calculate the difference between the cumulative decay values ​​at adjacent time points in the decay cumulative sequence, divide the difference by the time interval between adjacent time points to obtain the first gradient, and determine the gradient zero-crossing position based on the positive and negative changes of the first gradient. Gradient zero-crossing positions are arranged in chronological order, and adjacent gradient zero-crossing positions are used as segment boundaries. The cumulative decay value, gradient integral value, and duration within each segment are extracted to generate a temporally changing segment.

3. The disease risk prediction method based on multimodal data fusion according to claim 2, characterized in that, The process of performing exponential decay accumulation on the values ​​at each time point includes: Calculate the time difference between the current moment and each historical moment, divide the time difference by the median of the adjacent collection time intervals and take the negative value, and use the resulting value as the exponent of the natural constant to obtain the exponential decay. Multiply the historical values ​​by the corresponding exponential decay, accumulate the products, and add the accumulated result to the current value to obtain the cumulative decay value at the current time. The cumulative attenuation value is calculated sequentially according to the acquisition time, and the cumulative attenuation values ​​are arranged from earliest to latest to form the cumulative attenuation sequence corresponding to each data type.

4. The disease risk prediction method based on multimodal data fusion according to claim 1, characterized in that, The generation of multimodal disease evolution feature sequences includes: Nodes are divided according to temporal change segments. Cross-modal connections are established for nodes of different modalities with overlapping time ranges, and sequential connections are established for adjacent nodes of the same modality before and after time, thus constructing a modal channel node graph. Calculate the difference in cumulative attenuation, gradient integral, and duration between the two ends of each connection in the modal channel node graph. Normalize each difference and calculate the connection energy. Accumulate the connection energy layer by layer according to the node time order to obtain the cross-modal minimum energy path. The connection energy in the cross-modal minimum energy path is converted into path weights. The path weights are normalized and entropy is calculated. The obtained values ​​are corrected for row mean, column mean and overall mean to construct an entropy uniform matrix. The entropy uniform matrix is ​​divided into blocks, and singular value decomposition is performed on each block. The left singular vector matrix is ​​multiplied by the singular value matrix, and then the blocks are spliced ​​together according to the temporal change segments to generate a multimodal disease evolution feature sequence.

5. The disease risk prediction method based on multimodal data fusion according to claim 4, characterized in that, The method for obtaining the cross-modal minimum energy path includes: Calculate the difference in cumulative attenuation, gradient integral, and duration between the two connected nodes, and take the absolute value of the differences. Subtract the minimum difference of the current term from the absolute value of the difference, and then divide by the difference between the maximum and minimum differences to obtain the normalized difference. Square the normalized differences separately and add them together. Take the square root of the sum to obtain the connection energy. The calculation is performed layer by layer according to the node time sequence. The cumulative energy of the starting node is set to 0. The cumulative energy of the previous node is added to the connection energy to the current node to obtain the candidate energy. The minimum candidate energy is selected as the cumulative energy of the current node, and the corresponding previous node is recorded. After calculating the cumulative energy of each node in each layer, select the node with the smallest cumulative energy in the last layer, and trace back to the starting node layer by layer based on the previous node. Arrange the traced nodes in chronological order to obtain the cross-modal minimum energy path.

6. The disease risk prediction method based on multimodal data fusion according to claim 1, characterized in that, The obtained event-time joint probability matrix includes: An improved DeepHit model is constructed, which includes a shared subnetwork, a disease event-specific subnetwork, and an event-temporal joint probability output layer. A forward and reverse channel recursive calculation module is set between the shared subnetwork and the disease event-specific subnetwork, and a diagonal spectral difference folding module and a hierarchical residual backflow module are set after the disease event-specific subnetwork. The multimodal disease evolution feature sequence is input into the shared sub-network to extract shared risk features. The forward and reverse channel recursive calculation modules are used to perform forward time recursive calculation and reverse time recursive calculation respectively. The forward recursive calculation results, the reverse recursive calculation results, the difference between the two and the corresponding product are combined to obtain the disease event branch features. The disease event branch features are input into the disease event specific sub-network. The event time initial matrix is ​​calculated according to the disease event type and future time interval. The event time initial matrix is ​​continuously modified by the diagonal spectral difference folding module to obtain the event time correction matrix. The event time initial matrix and the event time correction matrix are added item by item and normalized. The single time interval probability, cumulative occurrence probability and non-onset probability are calculated through the hierarchical residual backflow module. The single time interval probability is corrected based on the cumulative back-calculation result to obtain the event time joint probability matrix.

7. The disease risk prediction method based on multimodal data fusion according to claim 6, characterized in that, The process of performing forward and reverse time recursion via the forward and reverse channel recursion modules includes: Read the shared risk features from early to late according to the temporal change segments, multiply the current shared risk feature by itself item by item to form an outer product matrix, and then recursively add it to the outer product matrix and feature accumulation vector of the previous time step to obtain the forward recursive calculation result; By recursively deriving shared risk characteristics from late to early according to the temporal change segments, the reverse external accumulation matrix and the reverse feature accumulation vector are obtained, forming the reverse recursive calculation result; Expand the forward and reverse recursive results separately, calculate the term-by-term difference and term-by-term product of the two, and then concatenate the forward and reverse recursive results, term-by-term difference and term-by-term product in sequence to obtain the disease event branch features.

8. The disease risk prediction method based on multimodal data fusion according to claim 7, characterized in that, The process of obtaining the forward recursive result includes: Set the external accumulation matrix and feature accumulation vector before the first time series change segment to 0, and read the shared risk feature vector corresponding to the current time series change segment; Each eigenvalue in the shared risk feature vector is multiplied by all eigenvalues, and the resulting products are arranged according to their feature positions to form the current outer product matrix; Add the current outer product matrix to the previous outer product matrix item by item, and add the current shared risk feature vector to the previous feature accumulation vector item by item to obtain the current outer product matrix and the current feature accumulation vector. The recursive processing is repeated sequentially according to the time-series change segments until the last time-series change segment. The external accumulation matrix and feature accumulation vector at each time point are arranged in chronological order to obtain the forward recursive calculation result.

9. The disease risk prediction method based on multimodal data fusion according to claim 1, characterized in that, The formation of the disease risk evolution sequence includes: Read the joint probability matrix of event time according to the disease event type and future time interval, and determine each matrix element as the probability of occurrence of the corresponding disease event in the corresponding future time interval; For each disease event, the probability of occurrence in the current time interval and all previous time intervals is accumulated in the order of future time intervals to obtain the cumulative probability of occurrence in each future time interval; Add up the cumulative probability of occurrence of each disease event within the same future time interval, and subtract the sum from 1 to obtain the probability of no disease occurrence in the corresponding future time interval; The cumulative probability of occurrence and the probability of non-occurrence of each disease event are arranged sequentially according to future time intervals. The difference in the cumulative probability of occurrence between adjacent future time intervals is calculated to form a disease risk evolution sequence.

10. A disease risk prediction system based on multimodal data fusion, comprising the disease risk prediction method based on multimodal data fusion as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The multimodal data processing module is used to collect multimodal medical data of the target object and perform preprocessing to obtain standard multimodal medical data; The temporal variation extraction module is used to perform exponential decay cumulative transformation and gradient zero-crossing detection on standard multimodal medical data sequences to form temporal variation segments; The evolutionary feature generation module is used to construct a modal channel node graph based on temporal change segments and generate a multimodal disease evolution feature sequence; The joint probability calculation module is used to input multimodal disease evolution feature sequences into the improved DeepHit model to obtain the event-temporal joint probability matrix; The risk evolution generation module is used to calculate the cumulative occurrence probability and non-occurrence probability of each disease event based on the event time joint probability matrix, forming a disease risk evolution sequence; The disease risk prediction module is used to determine the risk level and risk growth range of a target disease based on the disease risk evolution sequence, and generate disease risk prediction results. The risk path generation module is used to generate disease risk impact paths by taking the disease risk prediction results as the reverse starting point.

Images