Multimodal frequency domain enhancement method and device for predicting severe illness

Abstract

This application discloses a multimodal frequency domain enhanced critical illness prediction method and device. The method involves acquiring first and second data; extracting the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal; extracting static features from medical record data; fusing the static features with the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal to obtain a multimodal feature matrix; constructing a frequency enhancement prediction model; and inputting the multimodal feature matrix into the frequency enhancement prediction model to obtain the prediction results output by the frequency enhancement prediction model. By integrating multimodal time-series signals and static medical record data, and combining time-frequency analysis to fully capture the global frequency domain characteristics and local time domain details of the signal, this method effectively improves the accuracy and real-time performance of predicting the condition of critically ill patients, thereby providing reliable data support for clinical diagnosis and treatment decisions.

Description

Technical Field

[0001] This application relates to the field of medical artificial intelligence technology, and in particular to a multimodal frequency domain enhanced critical illness prediction method and device. Background Technology

[0002] Monitoring and predicting the level of consciousness (awake, comatose, or even death) of critically ill patients, especially those in neurocritical care, is a crucial aspect of clinical diagnosis and treatment, directly impacting treatment planning and prognostic assessment. Clinically, multimodal physiological signals such as electroencephalography (EEG), electrocardiography (ECG), and cerebral oxygen saturation (rSO2), combined with static information from electronic medical records, can comprehensively reflect a patient's brain function and overall physiological state, providing multidimensional data support for disease prediction. Therefore, intelligent prediction technology based on multimodal data fusion has become an important research direction in the field of intensive care.

[0003] However, existing technologies, through single-modal monitoring, have limitations in terms of information dimensions. They cannot fully reflect the correlation between brain function and physiological state, and are difficult to cover complex conditions. Moreover, the self-attention mechanism of traditional models such as Transformer has high computational complexity and limited ability to model non-stationary signals, resulting in insufficient accuracy in predicting the condition of critically ill patients, and thus making it difficult to effectively assist clinical decision-making. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] The main objective of this disclosure is to propose a multimodal frequency domain enhanced critical illness prediction method and device, which can effectively improve the accuracy and real-time performance of predicting the condition of critically ill patients by integrating multimodal time-series signals and static medical record data, and combining time-frequency analysis to fully capture the global frequency domain characteristics and local time domain details of the signals, thereby providing reliable data support for clinical diagnosis and treatment decisions.

[0006] A first aspect of this application provides a multimodal frequency domain enhanced critical illness prediction method, the method comprising:

[0007] Acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by sensors within the same time range; the second data includes the user's medical record data;

[0008] Extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal;

[0009] Extract static features from the medical record data;

[0010] By integrating the static features and the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal, a multimodal feature matrix is ​​obtained;

[0011] A frequency-enhanced prediction model is constructed; the frequency-enhanced prediction model is used to enhance the input of the frequency-enhanced prediction model in the frequency domain to predict the corresponding state probability;

[0012] The multimodal feature matrix is ​​input into the frequency enhancement prediction model to obtain the prediction result output by the frequency enhancement prediction model.

[0013] The multimodal frequency domain enhanced critical illness prediction method provided in this embodiment has at least the following beneficial effects:

[0014] This method acquires first and second data; extracts the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal; extracts static features from medical record data; fuses the static features with the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal to obtain a multimodal feature matrix; constructs a frequency enhancement prediction model; and inputs the multimodal feature matrix into the frequency enhancement prediction model to obtain the prediction results output by the frequency enhancement prediction model. By integrating multimodal time-series signals and medical record static data, and combining time-frequency analysis to fully capture the global frequency domain characteristics and local time domain details of the signal, this method effectively improves the accuracy and real-time performance of predicting the condition status of critically ill patients, thereby providing reliable data support for clinical diagnosis and treatment decisions.

[0015] To achieve the above objectives, a second aspect of the present invention provides a multimodal frequency domain enhanced critical illness prediction device, the device comprising:

[0016] The acquisition module is used to acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by sensors within the same time range; the second data includes the user's medical record data.

[0017] The extraction module is used to extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal;

[0018] The feature module is used to extract static features from the medical record data;

[0019] The fusion module is used to fuse the static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time-series signal to obtain a multimodal feature matrix;

[0020] A construction module is used to construct a frequency-enhanced prediction model; the frequency-enhanced prediction model is used to enhance the input of the frequency-enhanced prediction model in the frequency domain to predict the corresponding state probability;

[0021] The prediction module is used to input the multimodal feature matrix into the frequency enhancement prediction model and obtain the prediction results output by the frequency enhancement prediction model.

[0022] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the above-described multimodal frequency domain enhanced critical illness prediction method.

[0023] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described multimodal frequency domain enhanced critical illness prediction method.

[0024] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0026] Figure 1 This is a flowchart illustrating a multimodal frequency domain enhanced critical illness prediction method provided in an embodiment of this application;

[0027] Figure 2 This is a ROC curve of the model provided in this application embodiment on an internal dataset with respect to three state categories;

[0028] Figure 3 This is the ROC curve of the model provided in this application embodiment on an external dataset with respect to the three state categories;

[0029] Figure 4 This is a confusion matrix diagram of the model provided in this application embodiment on the internal dataset regarding the three state categories;

[0030] Figure 5 This is a confusion matrix diagram of the model provided in this application embodiment on an external dataset for the three state categories;

[0031] Figure 6 This is a comparison chart of the accuracy, precision, recall, and F1 score of the model provided in this application embodiment on internal and external datasets;

[0032] Figure 7 This is a schematic diagram of the structure of a multimodal frequency domain enhanced critical illness prediction device provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0035] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0036] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0037] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations. The acquisition, storage, use and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0039] Monitoring and predicting the level of consciousness (awake, comatose, or even death) of critically ill patients, especially those in neurocritical care, is a crucial aspect of clinical diagnosis and treatment, directly impacting treatment planning and prognostic assessment. Clinically, multimodal physiological signals such as electroencephalography (EEG), electrocardiography (ECG), and cerebral oxygen saturation (rSO2), combined with static information from electronic medical records, can comprehensively reflect a patient's brain function and overall physiological state, providing multidimensional data support for disease prediction. Therefore, intelligent prediction technology based on multimodal data fusion has become an important research direction in the field of intensive care.

[0040] However, the relevant technologies have limitations in terms of information dimensions due to single-modal monitoring. They cannot fully reflect the correlation between brain function and physiological state, and are difficult to cover complex conditions. Moreover, the self-attention mechanism of traditional models such as Transformer has high computational complexity and limited ability to model non-stationary signals, resulting in insufficient accuracy in predicting the condition of critically ill patients, and thus making it difficult to effectively assist clinical decision-making.

[0041] Based on this, the embodiments of this application provide a multimodal frequency domain enhanced critical illness prediction method and device, which aims to effectively improve the accuracy and real-time performance of predicting the condition of critically ill patients by integrating multimodal time-series signals and static medical record data, and combining time-frequency analysis to fully capture the global frequency domain characteristics and local time domain details of the signals, thereby providing reliable data support for clinical diagnosis and treatment decisions.

[0042] The multimodal frequency domain enhanced critical illness prediction method and apparatus provided in this application are specifically described through the following embodiments. First, the multimodal frequency domain enhanced critical illness prediction method in this application is described.

[0043] like Figure 1 As shown in one embodiment of this application, a multimodal frequency domain enhanced critical illness prediction method is provided, the method including the following steps S110 to S160.

[0044] Step S110: Acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by the sensor within the same time range; the second data includes the user's medical record data;

[0045] Step S120: Extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time series signal;

[0046] Step S130: Extract static features from medical record data;

[0047] Step S140: Fuse the static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time series signal to obtain the multimodal feature matrix;

[0048] Step S150: Construct a frequency enhancement prediction model; the frequency enhancement prediction model is used to enhance the input of the frequency enhancement prediction model in the frequency domain to predict the corresponding state probability;

[0049] Step S160: Input the multimodal feature matrix into the frequency enhancement prediction model to obtain the prediction results output by the frequency enhancement prediction model.

[0050] In this step, comprehensive and time-consistent multi-source data is acquired to lay the foundation for subsequent analysis. The first data source consists of multimodal time-series signals collected in real-time by sensors in intensive care equipment, including electroencephalogram (EEG), electrocardiogram (ECG), and cerebral oxygenation (CIO) signals. During acquisition, unified timestamps and synchronization markers ensure consistency across different signals over time, avoiding information bias caused by differences in sampling time. The second data source is medical record data from the hospital information system, encompassing non-time-series information such as basic patient information (e.g., age, gender), clinical diagnosis results, medication records, surgical history, and physician-annotated states of consciousness. This data reflects the patient's underlying health condition and treatment context, providing clinical contextual support for disease prediction.

[0051] Furthermore, to extract effective features from multimodal time-series signals, the signals need to be preprocessed before extraction, including denoising (such as bandpass filtering and wavelet thresholding), missing value imputation (linear interpolation and nearest neighbor imputation), and data standardization. For EEG signals, it is preferable to use a specialized recognition and removal model to remove physiological artifacts to ensure signal purity.

[0052] Specifically, feature extraction is achieved through two core transformations: Fourier transform is used to map the time-domain signal to the frequency domain to obtain the frequency domain amplitude spectrum (reflecting the signal energy distribution) and phase spectrum (revealing the coupling relationship between different modal signals) to capture the global frequency domain characteristics of the signal; continuous wavelet transform is used to expand the signal into multi-scale basis functions to extract multi-scale wavelet coefficients, accurately capturing the local time-domain details of non-stationary signals such as sudden EEG activity. The combination of the two transformations achieves comprehensive coverage of the global and local features of the signal.

[0053] For example, using Fourier transform and wavelet transform Comprehensive time-frequency feature extraction of physiological signals is performed, taking into account both global frequency domain characteristics and local time domain details, enabling the model to identify transient oscillations and long-term trends in EEG and ECG. Specifically, continuous wavelet transform is used to extract the signal... Mapping to scale Peaceful relocation The time-frequency plane, its transformation form is:

[0054] ;

[0055] in, For timestamps, Original signal, For the mother wavelet, Denotes the conjugate function. The transformed signal, parameters and This is a hyperparameter.

[0056] Furthermore, convolution operations are performed using multiple randomly generated convolution kernels to calculate statistical measures such as the maximum value and the proportion of positive values, forming the features enhanced by random convolution. Random convolution feature enhancement effectively expands the feature space. For example, a large number of random convolution kernels... Used with input sequence Perform convolution: For each convolution kernel, calculate the maximum value of the convolution result. and positive value proportion ,in Indicates taking The maximum value in the sequence, This represents the total length of the original sequence. For indicator functions, i.e. when The result of an operation is 1 if it is greater than 0, and 0 otherwise. These statistics form... 3D eigenvectors ( (This represents the total number of convolutional kernels). This random mapping eliminates the need to train convolutional kernels from the data and avoids overfitting.

[0057] Furthermore, structured static features are extracted from medical record data to provide clinical background information for multimodal fusion. During the extraction process, categorical variables (such as diagnosis type and whether surgery was performed) in the medical record data are quantified using one-hot encoding, and continuous variables (such as age and medication dosage) are normalized for mean and variance to eliminate dimensional differences. At the same time, to achieve adaptation with dynamic temporal features, key static features such as consciousness status labels are copied by time slice, so that the dynamic signals at each time point can correspond to the matching clinical status information, ensuring the consistency of static and dynamic features in the temporal dimension.

[0058] Furthermore, a hierarchical fusion strategy is used to integrate multi-dimensional features, forming a multimodal feature matrix that combines dynamic signal patterns with static clinical information. Specifically, local fusion is first performed by concatenating the frequency domain amplitude spectrum, phase spectrum, multi-scale wavelet coefficients, and features enhanced by random convolution of the same modality time-series signal to strengthen the expressive power of single-modality features. Then, global fusion is performed by concatenating the locally fused features of each modality with the preprocessed static features, and dynamically adjusting the contribution of each feature through a trainable weight vector. At the same time, a feature selection module filters highly relevant features and removes redundant information. The resulting multimodal feature matrix effectively compensates for the information limitations of single-modality data and achieves deep integration of multi-dimensional information.

[0059] Furthermore, a frequency-enhanced prediction model is constructed, specifically with a frequency domain enhancement module, a frequency domain attention module, and a hybrid expert decomposition module as its core, replacing the self-attention mechanism of the traditional Transformer: The frequency domain enhancement module projects the input sequence to the frequency domain through a linear transformation, utilizes the frequency sparsity of the time series signal to perform random sampling to retain effective components, and after completing the dimensions, transforms back to the time domain through an inverse transformation, significantly reducing computational complexity; The frequency domain attention module calculates the correlation between query, key, and value vectors in the frequency domain, performs attention weighting, and then transforms back to the time domain to strengthen the weight of effective signals; The hybrid expert decomposition module decomposes the input sequence into trend components (slow-changing trends) and periodic components (seasonal cycles), models them separately, and then reassembles them to improve the ability to capture multi-scale patterns. The three work together to achieve efficient and accurate time series modeling.

[0060] For example, the improved frequency-domain augmented timing model leverages frequency-domain sparsity to reduce the complexity of self-attention. The frequency-domain augmentation module (FEB module) takes the input sequence... Projected onto the frequency domain by linear transformation (in Given the input sequence, Let be the projection matrix. The transformed sequence is randomly sampled for frequency domain components and padded, then inversely transformed back to the time domain. This utilizes the sparsity of the time-series signal in the frequency domain, preserving useful information while reducing computational load. The frequency domain attention module (FEA) calculates attention weights in the frequency domain, including performing a Fourier transform to obtain... The frequency domain representation (where (Representing the query matrix, key matrix, and value matrix respectively), random sampling, and calculation. (in express The inverse matrix of the input sequence is obtained, the dimensions are filled, and the sequence is returned to the time domain via inverse Fourier transform. The hybrid expert decomposition block (MOEDecomp) decomposes the input sequence into trend components and periodic components. The decomposition formula can be expressed as follows: ,in For the seasonal cycle part, This refers to the long-term trend portion. The model... Frequency domain attention processing is performed, and Using residuals for reconstruction improves the ability to model multi-scale patterns. The overall model's time complexity is reduced by... Down to ,in The length of the time series is used to effectively improve the processing capability for long series and achieve good prediction results with a small amount of data.

[0061] Furthermore, disease status prediction is achieved by training a mature frequency-enhanced prediction model. Specifically, the training of the frequency-enhanced prediction model adopts a phased strategy: first, a classifier is trained based on the enhanced features to obtain initial weights, and then a frequency domain correlation module is incorporated for end-to-end fine-tuning. During the training process, a loss function suitable for multi-classification tasks is used, along with an adaptive optimizer, cosine annealing learning rate scheduling, regularization, and early stopping strategies to avoid overfitting and ensure generalization ability.

[0062] Specifically, in the prediction stage of the frequency enhancement prediction model, the constructed multimodal feature matrix is ​​input into the trained frequency enhancement prediction model. The frequency enhancement prediction model generates long-sequence prediction results in one go through the collaborative operation of internal modules, and outputs the patient's condition status category at a specified future time point (such as awake, coma, death), providing real-time and reliable decision-making basis for clinicians to formulate treatment plans and assess prognosis.

[0063] In one embodiment, a multimodal frequency domain enhanced critical illness prediction method is applied, firstly involving data acquisition and preprocessing. The specific steps are as follows:

[0064] 1. Multimodal data acquisition: Physiological indicators such as electroencephalogram (EEG), electrocardiogram (ECG), and cerebral oxygen saturation (rSO2) are acquired in real time from the monitoring system in the intensive care unit. The sampling frequencies are as follows: Simultaneously, the system retrieves the patient's basic information, medication records, surgical records, and the doctor's annotations on the patient's state of consciousness from the hospital's electronic medical record system. To ensure data consistency over time, the system uses a unified timestamp and synchronization marker.

[0065] Simultaneously, a deep learning model based on a convolutional neural network (CNN) was constructed for EEG signals. CNNs, with their unique structure of convolutional, pooling, and fully connected layers, can automatically extract hierarchical feature representations from input data. To train the model, a large amount of precisely labeled EEG data was collected. This labeled data was determined using professional medical knowledge and advanced signal analysis techniques, ensuring accurate identification of physiological artifacts in each data sample. During training, the model's parameters were adjusted using the backpropagation algorithm. Backpropagation is a gradient descent-based optimization method that calculates the gradient of the loss function relative to the model parameters and then updates the parameters in the opposite direction of the gradient to minimize the model's prediction error. Specifically, the cross-entropy loss function was used to measure the difference between the model's predictions and the actual labels. As training progresses, the model continuously learns the characteristic patterns of physiological artifacts, gradually improving its ability to identify and eliminate them.

[0066] 2. Signal Denoising and Slicing: Physiological signals often contain power frequency interference, electromyographic noise, and sensor drift. Therefore, a bandpass filter is used to denoise the ECG signal using wavelet thresholding; the cerebral oxygenation signal is smoothed using a moving average. Then, the signals are sliced ​​according to a fixed length. Each signal sequence is segmented so that each segment contains a uniform time step, denoted as . Indicates the first The first mode The segment corresponds to the first Data at each time step. Due to varying sampling lengths among different patients, zero-padding or linear interpolation was used to adjust the length of each segment to a fixed value.

[0067] 3. Missing Value Imputation and Normalization: If missing values ​​occur during the segmentation process, linear interpolation is used for consecutive missing segments, and nearest neighbor imputation is used for transient outliers. To eliminate the influence of different sensor dimensions, the Min-Max normalization method is used for each feature, and its expression is as follows:

[0068] ;

[0069] in, These are the original eigenvalues. This is the maximum value of the feature across all samples. These are the minimum values ​​of the feature in all samples. This represents the normalized value.

[0070] 4. Static Feature Engineering: Electronic medical records contain non-time-series information such as patient age, gender, diagnosis type, medication use, and surgical procedures. Categorical variables are encoded using one-hot encoding, and continuous variables are normalized for mean and variance. To describe changes in clinical status over time, some static features (such as consciousness status labels) are replicated at each time slice, creating a one-to-one correspondence with dynamic features.

[0071] Furthermore, multimodal feature extraction and fusion are performed on the aforementioned multimodal data. First, Fourier transform feature extraction is performed on each segmented time-series signal sequence. The Discrete Fourier Transform (DFT) is expressed as follows:

[0072] ;

[0073] ;

[0074] in, The time-series signal after Fourier transform. For the first angular frequency, The time window length, For the index of sampling points within the time window, It is the imaginary unit.

[0075] Specifically, the Fourier transform maps the time-domain signal to the frequency domain, obtaining the amplitude spectrum. and phase spectrum The amplitude spectrum reflects the energy distribution of the signal, while the frequency components correspond to the intensity of the brainwave. Different frequency bands such as waves; phase spectrum reveals the phase difference between different modes, which helps to explore the coupling relationship between EEG and ECG.

[0076] Furthermore, since non-stationary signals such as sudden EEG activity have transient characteristics on the time axis, Fourier transform is insufficient to capture local features. Therefore, continuous wavelet transform (CWT) is used to expand the signal into basis functions of different scales and locations, as expressed below:

[0077] ;

[0078] in, The time-series signal after wavelet transform. Frequency is controlled by scale parameters. The translation parameters control the time and position. For the mother wavelet function, The time variable is used. Mexican Hat or Morlet wavelets are commonly chosen as the mother wavelet. Different scales correspond to different frequency bands; higher scales correspond to lower frequencies, and lower scales correspond to higher frequencies. The magnitude of the wavelet coefficients reflects the signal strength at a specific time and frequency band. The inverse wavelet transform can reconstruct the original signal, and its expression is as follows:

[0079] ;

[0080] in, The original signal after inverse wavelet reconstruction. Frequency is controlled by scale parameters. The translation parameters control the time and position. For time variables, The differential symbol, Let be the acceptability constant. yes The dual wavelet is then used. Furthermore, by statistically analyzing the wavelet coefficients, the energy, entropy, mean, and variance at each scale are calculated to extract transient events and abrupt changes in EEG and ECG.

[0081] Furthermore, to further enhance feature representation capabilities, after extracting Fourier and wavelet features, a random convolution kernel transformation is applied to each time segment. Let there be a total of... random convolution kernels ,in, For convolution kernel, The convolution sequence number is used, and the length, dilation rate, and weights of each convolution kernel are randomly generated. For the input sequence... The expression for performing a one-dimensional convolution operation is as follows:

[0082] ;

[0083] in, The time slice data after one-dimensional convolution. The convolution index is... For sequence length, The input sequence is then used. The convolution result is then... The two statistics are calculated as follows:

[0084] Maximum value: This reflects the maximum response of the pattern corresponding to the convolution kernel in the sequence;

[0085] Positive proportion: ;

[0086] in, The length of the convolution output. This is an indicator function. The positive proportion measures the proportion of convolutional kernels that are positively activated, reflecting the frequency of pattern occurrence.

[0087] Furthermore, for each convolution kernel, we obtain Two-dimensional features Each convolutional kernel generates Dimensional vector. These features are combined with the previous frequency domain features and static features to form a comprehensive feature matrix. ,in The total dimension of the features is given. Then, probability density analysis is performed on the convolution results, calculating the mean of the convolution output. and standard deviation We then standardize it, and the expression is as follows:

[0088] ;

[0089] in, These are the normalized eigenvalues. The mean, The standard deviation is used to compare the relative contributions of different convolutional kernels. Furthermore, mutual information or discrete Fourier analysis can be used to filter out the most discriminative convolutional features, further reducing dimensionality.

[0090] Furthermore, to more comprehensively describe the statistical properties of multimodal physiological signals, a series of theoretical indices and statistics were introduced to capture subtle differences in the spectrum and time series. These include:

[0091] (1) Energy spectral density. Amplitude spectrum of discrete Fourier transform. It can be used not only to calculate amplitude, but also to derive the energy spectral density of a fragment:

[0092] ;

[0093] in, () represents the time-series signal after Fourier transform. For the first angular frequency, The time window length, Let be the energy spectral density corresponding to the segment. Furthermore, according to Parseval's theorem, the energy of a signal in the time domain is equal to the sum of its energy in the frequency domain, as expressed below:

[0094] ;

[0095] in, () represents the time-series signal after Fourier transform. For the first angular frequency, The time window length is used to analyze the energy distribution of different frequency bands through energy spectral density analysis, which is of great significance for identifying changes in brain electrical rhythm and variability in electrocardiogram heart rate.

[0096] (2) Spectral Centroid and Spectral Divergence. The spectral centroid measures the center of gravity of the signal energy distribution, and is defined as follows:

[0097] ;

[0098] ;

[0099] in, For the first The physical frequency corresponding to each frequency point Sampling rate, The centroid of the energy distribution corresponding to the signal. () represents the time-series signal after Fourier transform. For the first angular frequency, The time window length, This is the slice number. Furthermore, spectral divergence describes the degree of energy dispersion along the frequency axis, and its definition is as follows:

[0100] ;

[0101] ;

[0102] in, For spectral divergence, The mean of the sequence. For the first The physical frequency corresponding to each frequency point Sampling rate, The time window length, This is the fragment number. This is the amplitude spectrum. Moreover, in a coma, the energy center of gravity of brain waves is generally low and the dispersion is large; while in a conscious state, the increase in high-frequency components causes the center of gravity to shift upward.

[0103] (3) Spectral entropy and information content. To measure the disorder of the spectral distribution, spectral entropy is introduced, and its expression is as follows:

[0104] ;

[0105] ;

[0106] in, For the normalized amplitude spectrum, For spectral entropy, The time window length, For the ordinal variable, This is the fragment number. This refers to the amplitude spectrum. A higher spectral entropy indicates a more uniform energy distribution and a signal that is closer to a random process; a lower spectral entropy indicates that frequency components are concentrated in a few peaks. Spectral entropy can be used to distinguish between brain electrical activity in conscious and anesthetized states.

[0107] (4) Cross-correlation function and phase synchronization. To explore the synchronization relationship between different modes, the mode can be calculated. and The cross-correlation function between them is expressed as follows:

[0108] ;

[0109] in, for Time series The signal value, for Time series The signal value, For modality and The cross-correlation function between them The time window length is determined by searching... The maximum value and its corresponding The delay between two signals can be estimated. Simultaneously, the phase synchronization index can also be calculated by statistically analyzing the instantaneous phase difference between the two signals. The distribution is calculated, and its expression is as follows:

[0110] ;

[0111] in, The phase synchronization index of the two signals is given. It is an exponential function. The instantaneous phase difference between the two signals. This represents the length of the time window. Phase synchronization indicators help reveal the impact of heart-brain coupling and brain-oxygen coupling on states of consciousness.

[0112] (5) Wavelet energy and multi-scale entropy. The coefficients of the continuous wavelet transform... Energy can be calculated at each scale, and its expression is as follows:

[0113] ;

[0114] in, For wavelet energy, These are continuous wavelet transform coefficients. To Perform integration and define the scale probability density. Based on this, the multi-scale entropy can be calculated, and its expression is as follows:

[0115] ;

[0116] in, For multi-scale entropy, This represents the joint probability distribution among signals, while Indicates the signal Integrating this indicator reveals its strong medical significance: high-scale (low-frequency) wavelet entropy reflects the complexity of long-period activities, while low-scale (high-frequency) entropy reflects the changes in transient activities. Combining energy and entropy features at different scales allows for the construction of wavelet energy spectra, which are then used as input for subsequent models. Furthermore, by adding the aforementioned theoretical indicators, this embodiment considers not only the signal's energy distribution, centroid, entropy, and cross-correlation at the feature level, but also the energy and complexity at multiple scales, providing richer input information for deep time-series models.

[0117] Furthermore, a hierarchical fusion strategy is adopted. First, local fusion is performed. Within each time slice, the frequency domain features, wavelet features, and random convolution features of EEG, ECG, and rSO2 are concatenated according to their dimensions to obtain a dynamic feature vector. ,in, For time steps, This represents the length of the feature vector. For multiple EEG channels, principal components can be extracted using channel attention or principal component analysis, and then merged with ECG and rSO2 features. Then, global fusion is performed to combine the dynamic features. Static features Serial connection To consider the importance of different features, a trainable weight vector is introduced. The fusion result is ,in This represents the Hadamard product. Where, Through learning during training, the model can automatically adjust the contribution of each modality feature. Finally, feature selection is performed, using mutual information, correlation coefficients, or attention-based feature selection modules to filter out the features most relevant to the prediction task, reducing redundancy. The feature selection module can be represented by a gating mechanism. ,in For trainable parameters, This is the Sigmoid function. The resulting fused feature sequence... As input to the improved time series model.

[0118] Furthermore, the weights learned through training The model dynamically reweights the contributions of different modal features: it automatically assigns higher weights to modalities that contain key symptoms at the current time step, while suppressing modalities with poor signal quality or those subject to interference, thus ensuring the quality of the generated feature sequences. It is a high-quality feature representation with importance markers.

[0119] Furthermore, after obtaining the fused feature sequence through the aforementioned feature fusion and screening, considering that the multimodal physiological signals (such as EEG and ECG) of critically ill patients often contain complex periodic variation patterns, and that processing methods based directly on one-dimensional sequences are insufficient to fully capture these long-range dependencies across periods, this application introduces the following two-dimensional periodic transformation and multi-scale convolution module (based on multi-period temporal learning) after the feature fusion layer to further enhance the model's ability to perceive multi-scale periodic patterns and provide a more structured feature representation for subsequent frequency enhancement prediction. Its core process includes:

[0120] 1. Periodic Detection and Rearrangement: For the input multimodal feature matrix First, perform a Fast Fourier Transform on each feature dimension and calculate the frequency. average amplitude Choose the one with the largest amplitude. Frequency components The corresponding period length is According to each cycle Rearrange a one-dimensional sequence into a two-dimensional tensor. In a two-dimensional tensor, columns represent time steps within a period, and rows represent time steps between adjacent periods.

[0121] 2. Multi-scale two-dimensional convolution: For each period tensor Feature extraction is performed using parameter-efficient multi-scale two-dimensional convolutional kernels. These methods capture short-period and long-period patterns respectively. The convolution operation is defined as:

[0122] ;

[0123] in, To indicate the first The convolution scale, the first The output features of the periodic tensor at position The value, For this is the first The height of each convolutional kernel, This represents the index of the convolution kernel in the height direction. This represents the index of the convolution kernel along its width. For the first The width of each convolutional kernel, It is a non-linear activation function. For this is the first The periodic tensor at position The value, The bias coefficient, For the maximum height of the core, The maximum width of the kernel. By computing convolutional kernels of different sizes in parallel, local and global patterns both within and across periods can be captured simultaneously.

[0124] 3. Feature Reconstruction and Fusion: For each cycle The convolution result is flattened along the periodic dimension to obtain a one-dimensional representation. According to the corresponding magnitude weight. The weighted fusion is expressed as follows:

[0125] ;

[0126] ;

[0127] in, The characteristics after fusion The total number of cycles. For the first Weights for each period, For the first A one-dimensional representation of a period For the first Frequency parameters per cycle, For amplitude weighting function, This is the cycle number. Thus, this amplitude-based weighted fusion reflects the importance of each cycle, and the fusion result... As a more semantically and periodically informative representation, it is then input into a frequency domain-enhanced attention module for modeling. This module effectively decouples variation patterns of different period lengths through period detection, two-dimensional rearrangement, and multi-scale convolution, and adaptively aggregates the period representations through amplitude weighting, thereby providing richer input features for subsequent time-series models.

[0128] Furthermore, a frequency-domain enhanced time-series model is constructed. Specifically, a frequency-domain block-based approach is used to replace part of the attention computation, assuming the input sequence is... Frequency domain enhancement block (FEB) first... Projected into the frequency domain:

[0129] ;

[0130] ;

[0131] ;

[0132] in, Let be a trainable linear transformation matrix. This represents the Fast Fourier Transform. Because time-series signals are sparse in the frequency domain, they can be randomly sampled and preserved in the frequency domain. Each frequency component is denoted as the sampling set. Random sampling can employ Top-K energy or uniform sampling. The resulting representation... Used for subsequent attention calculations, reducing the computational cost to [missing information]. Then, the frequency domain dimension is padded with zeros and an inverse FFT is performed:

[0133] ;

[0134] in, This indicates that the sampled results will be padded to the length. , This is the inverse Fourier transform. Due to the use of a low-rank approximation and sparse sampling in the frequency domain, the FEB significantly reduces computational complexity. The FEB can also use wavelet bases for frequency domain representation, making it particularly suitable for non-stationary signals. The input sequence is processed through a wavelet decomposition matrix... Projection to multi-scale representation: ,in, Given the input sequence, The wavelet decomposition matrix is... The sequence after wavelet decomposition is processed by processing the high-frequency, low-frequency, and residual components separately, and then the reconstruction matrix is ​​used. Returning to the time domain.

[0135] Furthermore, based on FEB, the Frequency Domain Attention Module (FEA) implements the attention mechanism in the frequency domain. Firstly, regarding... Perform FFT: Then, the frequency components are randomly sampled and the attention weights are calculated:

[0136] ;

[0137] in, For complex conjugate, This is a pointwise product. After padding the dimensions, an inverse FFT is performed to obtain the time-domain output. If a wavelet basis is selected, the FFT is replaced with a wavelet transform; for non-stationary signals, wavelet domain attention can better handle abrupt changes and edge information.

[0138] Furthermore, a hybrid expert decomposition (MOEDecomp) module is introduced to decompose the trend and seasonality terms in the input sequence:

[0139] ;

[0140] in, Represents the periodic part, This represents a slow-changing trend. MOEDecomp utilizes a set of expert networks to model the seasonal and trend components separately. For the seasonal component, frequency domain attention aligns the periodic patterns, while for the trend component, gated recurrent units or linear layers are used for modeling. The decoder stage then combines these components to reconstruct the output sequence. This decomposition helps alleviate pattern aliasing and improves prediction accuracy.

[0141] Furthermore, in multimodal scenarios, there are potential coupling relationships between different sensors. For example, changes in brain electrical activity may be affected by heart rate fluctuations, and brain oxygen fluctuations may be synchronized with ECG abnormalities. This embodiment introduces a cross-modal interaction layer after two-dimensional periodic transformation to explore these correlations. Let the rearranged representation be... ,in This represents the number of modalities (e.g., EEG channels, ECG, rSO2). The cross-modal interaction layer calculates the correlation matrix between channels. Its elements are defined as:

[0142] ;

[0143] in, For modality With mode The degree of cosine similarity in the periodic dimension and the dimension within the period is located in the interval (-1, 1). For the first The total number of features for each modality For the first The total number of features for each modality For the ordinal variable, For the ordinal variable, For the first The sequence number of each mode. For the first The sequence number of each mode. Indicates the first The Frobenius norm of each modality over the entire two-dimensional tensor.

[0144] Furthermore, according to Constructing modal attention weights This layer is used to weight convolutional features from different modalities, thereby capturing the collaborative changes between modalities early in the model and preventing subsequent networks from ignoring weak but important correlation signals. By combining with random convolution and frequency domain enhancement modules, the cross-modal interaction layer further improves the fusion effect and prediction accuracy of multimodal data.

[0145] Furthermore, a generative decoder is employed, utilizing the global representation output by the encoder to generate long-sequence predictions in a single step, thus avoiding accumulated errors. The decoder consists of multiple stacked FEAs and fully connected layers, with an output dimension equal to the target prediction length. Then, the probability distributions for each category are obtained through a Softmax layer, and their expressions are as follows:

[0146] ;

[0147] in, This is the final hidden state of the decoder. For output layer parameters, This represents the probability value of the patient's predicted category, as determined by the model. Based on... The maximum component determines the patient's state of consciousness category (awake, comatose, or dead).

[0148] Specifically, for the training and optimization of the frequency domain enhanced time series model, the cross-entropy loss function is adopted, and its expression is as follows:

[0149] ;

[0150] in, The loss function of the model. For the sample size, The sample number. The total number of categories, For category number, It is a unique hot code for the real label. Predict probabilities for the model. For example, This represents three categories of consciousness states. To prevent overfitting, a factor is added to the loss function. Regularization and dropout.

[0151] Furthermore, the model employs a staged training strategy. First, a linear classifier is trained using random convolutional features to obtain initial weights; then, a frequency domain enhancement block and an attention module are added for end-to-end fine-tuning of the model. The Adam optimizer is used, with cosine annealing scheduling for the learning rate and a batch size set to [value missing]. During training, early stopping is used to monitor the validation set loss and avoid overfitting. Accuracy is then used as the benchmark. Accuracy Recall rate The model's performance on the test set is evaluated using metrics such as the F1 score. For multi-class classification tasks, these metrics can be expressed using either macro-average or weighted average.

[0152] Furthermore, a series of strategies and hyperparameter settings were adopted, including:

[0153] 1. Optimizer and Learning Rate Scheduling: The Adam optimizer is selected, whose adaptive learning rate mechanism balances convergence speed and stability. The initial learning rate is set to... The learning rate is gradually reduced using cosine annealing, as expressed below:

[0154] ;

[0155] ;

[0156] in, For real-time learning rate, To minimize the learning rate, This represents the total number of training rounds. This scheduling helps prevent the learning rate from dropping too quickly and causing training stagnation.

[0157] 2. Batch size and gradient pruning: Set the batch size according to the video memory and data size. After calculating the gradient, gradient clipping is applied to control the gradient norm from not exceeding a threshold. Gradient pruning can alleviate the gradient explosion problem, which is especially important in deep networks and long sequences.

[0158] 3. Weight Initialization and Regularization: Linear layer weights are initialized using Xavier uniform initialization, and convolutional kernels are initialized using He; bias terms are initialized to zero. L2 regularization (weight decay coefficient) is used during training. And Dropout, to prevent overfitting.

[0159] 4. Early stopping strategy: Monitor cross-entropy loss and F1 score on the validation set, and stop training early when the validation loss does not decrease for 5 consecutive epochs. This can avoid overfitting on the training set and improve the model's generalization ability.

[0160] 5. Cross-validation: To make full use of the limited clinical sample, five-fold cross-validation is used to divide the data into training and validation sets. After each fold, the average performance index is calculated, and the final model's hyperparameters are selected based on the average performance of the validation set.

[0161] 6. Monitoring signal balance: Due to the uneven distribution of the patient's three states of consciousness, this invention adds class weights to the loss function. This is to mitigate the impact of class imbalance on the model.

[0162] Therefore, through the above training strategies and reasonable hyperparameter selection, the model can converge stably on limited data and maintain high accuracy and generalization ability on the test set.

[0163] Specifically, validation was performed using multimodal data from 120 critically ill patients collected at a hospital. The dataset was divided into training, validation, and test sets in a 6:1:3 ratio. Specific experimental results are as follows: Figure 2 , Figure 3 , Figure 4 , Figure 5 as well as Figure 6 As shown, where, Figure 2 The ROC curves (Receiver Operating Characteristic curves) for the model on the internal dataset for three state categories (awake, comatose, and dead) are shown. The horizontal axis represents the false positive rate, and the vertical axis represents the true positive rate. Predictive performance for each category is as follows: the AUC for "Dead" (Category 2, solid green line) is 1.00; the AUC for "Awake" (Category 0, solid blue line) is 0.95; and the AUC for "Comatose" (Category 1, solid orange line) is 0.81, exhibiting a stepped shape. The overall average AUC of the model is 0.93 (dashed pink line), and all classification curves lie above the random guess diagonal.

[0164] Specifically, Figure 3 The ROC curves for the model on an external dataset for the three state categories (awake, comatose, and dead) are shown (where the horizontal and vertical axes represent the false positive and true positive rates, respectively). The model performed stably on the external data validation, with an average AUC of 0.92 (pink dashed line). The performance for each category is as follows: the model's predicted AUC for "dead" (category 2, green solid line) remained at 1.00; the predicted AUC for "comatose" (category 1, orange solid line) was 0.88, showing some improvement compared to the internal validation results; while the predicted AUC for "awake" (category 0, blue solid line) was 0.79, indicating relatively low generalization performance for this category on the external data. Overall, all classification curves lie above the random guess diagonal, validating the model's effectiveness.

[0165] Specifically, Figure 4 The confusion matrix for the model on the internal dataset for the three state categories (awake, comatose, and dead) is shown in the figure. This figure is used to evaluate the model's classification accuracy at a specific threshold in detail. The overall metrics show that the model's accuracy is 0.80 and the F1 score is 0.75. In terms of specific performance for each category: the model has the strongest recognition ability for "comatose" (category 1), with all 6 actual samples correctly predicted (value 6 in the center of the matrix); the prediction for "awake" (category 0) is mostly correct, with 2 out of 3 actual samples correctly identified and 1 misclassified as "comatose"; while for "dead" (category 2), the only sample in the validation set was not identified at this decision threshold (value 0 in the lower right corner) and was misclassified as "comatose". The data shows that all misclassified samples were classified as "comatose", indicating that the model tends to predict "comatose" for uncertain samples under the current parameter settings.

[0166] Specifically, Figure 5The confusion matrix of the model on the external dataset for the three state categories (awake, comatose, and dead) is shown in the chart. The statistical indicators indicate that the model accuracy is 0.90 and the F1 score is 0.89. Specifically, the sample distribution shows that the model's recognition performance for "comatose" (category 1) is the most stable, with all 6 actual samples correctly predicted. For "awake" (category 0), 2 out of 3 samples were correctly predicted, and 1 was misclassified as "comatose." For "dead" (category 2), 1 sample in the validation set was not correctly identified and was misclassified as "comatose." The data shows that all classification errors in the external validation were due to misclassifying samples of other categories as "comatose."

[0167] Specifically, Figure 6 This chart compares the model's accuracy, precision, recall, and F1 score on both internal and external datasets. It shows a comparison of key performance indicators (APIs) on the internal validation set (blue bars) and the external validation set (orange bars), covering four dimensions: accuracy, precision, recall, and F1 score. The data comparison shows that the model's APIs on the external validation set are significantly higher than those on the internal validation set: accuracy increased from 0.80 internally to 0.90 externally; precision saw the most significant improvement, increasing from 0.75 to 0.91; recall improved from 0.80 to 0.90; and the F1 score, reflecting the model's robustness, also increased from 0.75 to 0.89. This result indicates that the model demonstrates excellent generalization ability on the current external test data, with substantial improvements in overall prediction performance across all dimensions.

[0168] As shown in Table 1, Table 1 presents the overall performance of the model on the internal validation set:

[0169] Table 1

[0170]

[0171] The detailed classification report is shown in Table 2:

[0172] Table 2

[0173]

[0174] As shown in Table 3, Table 3 presents the overall performance of the model on the external validation set:

[0175] Table 3

[0176]

[0177] The detailed classification report is shown in Table 4:

[0178] Table 4

[0179]

[0180] Furthermore, the experiments revealed that increasing the number of random convolutional kernels (M) from 5000 to 10000 significantly improved the model's accuracy; the Mexican Hat wavelet performed slightly better than the Morlet wavelet in wavelet transform; and the model using frequency domain attention reduced the error in long sequence prediction by approximately 5% compared to the pure time domain attention model. Additionally, adjusting the sampling ratio (M / N) allowed for a trade-off between accuracy and speed. A sampling ratio of 0.25 resulted in a significant improvement in model speed with minimal accuracy loss.

[0181] Therefore, this embodiment achieves accurate prediction of the condition of critically ill patients by combining multimodal data fusion, random convolution feature enhancement, and frequency domain enhanced time series model. Specifically, Fourier and wavelet transforms are used to perform time-frequency analysis on the signal, enabling the model to capture both global spectral information and identify local transient events; random convolution features further expand the feature space, enhancing the model's ability to represent complex patterns. Moreover, through frequency domain projection, random sampling, and inverse transform, the complexity of self-attention is reduced from quadratic to near linear, significantly improving the efficiency of long sequence processing, making it suitable for real-time monitoring applications. In addition, the random convolution kernel in this embodiment does not rely on data learning, avoiding overfitting; hybrid expert decomposition enables the model to simultaneously model trend and seasonal patterns, exhibiting good generalization performance for different types of sequences. Furthermore, each module in this embodiment (preprocessing, feature extraction, convolution feature enhancement, and frequency domain attention) can be independently replaced or upgraded, facilitating adjustment and optimization according to specific task requirements. This provides a new solution for intelligent medical monitoring and early warning systems, possessing high academic value and broad application prospects.

[0182] In some embodiments, step S120 involves extracting the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal, including the following steps S210 to S230:

[0183] Step S210: The EEG signal, ECG signal, and brain oxygenation signal are segmented to obtain the first signal, the second signal, and the third signal, respectively.

[0184] Step S220: Perform Discrete Fourier Transform on the first signal, the second signal, and the third signal respectively to obtain the frequency domain amplitude spectrum, the phase spectrum of the first signal, the frequency domain amplitude spectrum, the phase spectrum of the second signal, the frequency domain amplitude spectrum, and the phase spectrum of the third signal.

[0185] Step S230: Perform wavelet transform on the first signal, the second signal and the third signal respectively to obtain the multi-scale wavelet coefficients of the first signal, the multi-scale wavelet coefficients of the second signal and the multi-scale wavelet coefficients of the third signal.

[0186] In this embodiment, the continuously acquired EEG, ECG, and cerebral oxygenation signals are first converted into segments of uniform length and time steps. During the segmentation process, a unified timestamp is used as a reference to ensure that each segment contains an equal number of time steps. Specifically, for signals with a sampling length shorter than the fixed segment length, zero-padding or linear interpolation is used to supplement the length; for signals exceeding the fixed length, continuous segmentation is performed according to a set step size to avoid information loss. The resulting first signal (after EEG signal segmentation), second signal (after ECG signal segmentation), and third signal (after cerebral oxygenation signal segmentation) maintain consistency in the time dimension, and each segment can independently reflect the physiological signal change characteristics within the corresponding time period, providing a standardized data foundation for subsequent synchronous extraction of frequency domain features.

[0187] Furthermore, by using the Discrete Fourier Transform (DFT), the three types of time-domain signals (first, second, and third signals) after being segmented are mapped to the frequency domain, enabling the extraction of the global frequency domain characteristics of the signal. The core function of the DFT is to decompose the frequency components of the signal, transforming the time-domain signal, which originally varies with time, into a superposition of different frequency components.

[0188] Specifically, for the first signal (EEG slice signal), the transformed frequency domain amplitude spectrum clearly reflects the energy distribution of different frequency bands of the EEG (such as slow waves and fast waves), intuitively demonstrating the energy characteristics of neuronal activity in the brain; the phase spectrum reveals the phase relationship between the frequency components of the EEG signal, helping to uncover the coupling patterns between EEG and other modal signals. Similarly, the frequency domain amplitude spectrum of the second signal (ECG slice signal) reflects the frequency components and energy changes related to heart rate, while the phase spectrum assists in analyzing the rhythmic synchronicity of the ECG signal; the frequency domain spectral characteristics of the third signal (brain oxygen slice signal) reflect the frequency fluctuation patterns related to brain oxygen metabolism. Through this step, the global frequency characteristics of the three types of signals are quantified into computable frequency domain amplitude and phase spectra, providing core information in the frequency domain dimension for subsequent multimodal feature fusion.

[0189] Furthermore, wavelet transform is employed to process the three types of segmented signals, focusing on capturing local time-domain details and non-stationary characteristics, complementing the global frequency-domain analysis of discrete Fourier transform. Since sudden abnormal activity in EEG signals and instantaneous rhythmic fluctuations in ECG signals are non-stationary, traditional Fourier transform struggles to accurately capture their local changes. Wavelet transform, by decomposing the signal into basis functions (mother wavelets) at different scales and locations, enables joint time-frequency domain analysis. During processing, suitable mother wavelets (such as Mexican hat wavelets or Morlet wavelets) are selected for the first, second, and third signals respectively. The analysis frequency is controlled by adjusting the scale parameter (high scale corresponds to low frequency, low scale corresponds to high frequency), and the time position is located by shifting the parameter, thus obtaining multi-scale wavelet coefficients. The magnitude of these coefficients directly reflects the signal intensity at a specific time point and frequency band, accurately capturing local features such as sudden EEG oscillations and instantaneous ECG abnormalities. Wavelet coefficients at different scales correspond to the low-frequency trend and high-frequency details of the signal, respectively, comprehensively covering the multi-scale variation law of the signal, and providing rich local time-domain-frequency domain joint features for subsequent feature enhancement and fusion.

[0190] In some embodiments, in step S140, the static features and the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time series signal are fused to obtain a multimodal feature matrix, including the following steps S310 to S330:

[0191] Step S310: Perform random convolution kernel transformation on the static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time series signal to obtain the enhanced static features, the enhanced frequency domain amplitude spectrum, the enhanced phase spectrum and the enhanced multi-scale wavelet coefficients corresponding to each modal time series signal.

[0192] Step S320: Fuse the enhanced frequency domain amplitude spectrum, enhanced phase spectrum and enhanced multi-scale wavelet coefficients corresponding to each modal time series signal to obtain the multi-modal characteristic vector corresponding to each modal time series signal;

[0193] Step S330: Fuse the enhanced static features and the multimodal characteristic vectors corresponding to each modal time-series signal to obtain the multimodal feature matrix.

[0194] In this embodiment, the feature space is expanded and the feature representation capability is enhanced by random convolution kernel transformation, while avoiding the risk of overfitting, thus providing a richer high-dimensional feature foundation for subsequent fusion. Specifically, transformations are performed on two types of core data: one is the static features extracted from medical record data, and the other is the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients obtained from time-frequency analysis of various modal time-series signals.

[0195] Specifically, during the transformation process, a large number of random convolution kernels are generated. The length, dilation rate, and weights of each kernel are randomly determined, eliminating the need for training from data. This simplifies the computation process and avoids dependence on specific data distributions. After performing a one-dimensional convolution operation on each type of input feature, two key statistics are calculated: the maximum value of the convolution result, reflecting the strongest response of the feature pattern corresponding to that kernel in the input sequence; and the positive proportion, measuring the frequency of the pattern's occurrence in the sequence. By extracting these statistics, the original features are transformed into high-dimensional enhanced features, preserving not only the core information of the original features (such as frequency domain energy distribution and local temporal details) but also uncovering potential latent patterns, effectively improving the discriminative power of the features. Furthermore, the randomness of the convolution kernels reduces the possibility of overfitting.

[0196] Furthermore, the enhanced features of this modality are uniformly aggregated to form a multimodal characteristic vector that comprehensively represents the information of this modality. For each type of modal time-series signal (such as EEG signals, ECG signals, and brain oxygenation signals), its enhanced frequency domain amplitude spectrum (reflecting the global frequency energy distribution), enhanced phase spectrum (revealing the coupling relationship between frequency components), and enhanced multi-scale wavelet coefficients (capturing local time-domain-frequency domain joint details) are spliced ​​and fused according to the feature dimensions. This fusion method can deeply integrate features of different dimensions (global and local, frequency domain and time domain, original and enhanced) under the same modality, avoiding the information limitations of a single feature dimension, and forming a comprehensive characteristic vector specific to this modality. For example, the characteristic vector of an EEG signal will simultaneously include its enhanced frequency domain energy features, phase correlation features, multi-scale local abnormal features, and statistical features enhanced by random convolution, comprehensively depicting the complex laws of brain neuron activity, and providing a single-modal basic feature with a unified structure and complete information for cross-modal fusion.

[0197] Furthermore, a hierarchical fusion strategy is employed to organically combine static clinical information with dynamic features from various modalities, ultimately forming a multimodal feature matrix that is both comprehensive and targeted. This matrix encompasses the dynamic changes of multimodal time-series signals while also incorporating the patient's static clinical background, achieving comprehensive information coverage of "dynamic signals + static medical records." The fusion process follows the logic of "first integrating within a single modality, then integrating across modalities and static data."

[0198] Specifically, the multimodal feature vectors corresponding to the temporal signals of each modality are first initially integrated to ensure temporal consistency and information complementarity among dynamic signals. Then, the integrated dynamic feature set is concatenated with enhanced static features (such as clinical background information like patient age, diagnosis type, and medication history). To accommodate the varying importance of different features, a trainable weight vector is introduced during the fusion process. Model training adaptively adjusts the contribution of dynamic and static features, as well as features from different modalities, giving higher weight to key information (such as abnormal frequency domain features of EEG signals and critical illness-related diagnostic information). Simultaneously, a feature selection module (based on mutual information, attention mechanisms, etc.) filters out features highly relevant to disease prediction, eliminating redundant information, ultimately forming a multimodal feature matrix with reasonable dimensions and dense information.

[0199] In some embodiments, the frequency enhancement prediction model outputs the prediction result in step S150 through the following steps S410 to S460:

[0200] Step S410: Perform frequency domain enhancement on the multimodal feature matrix to obtain the first feature matrix;

[0201] Step S420: Calculate the vector correlations corresponding to the query matrix, key matrix, and value matrix of the first feature matrix in the frequency domain of the first feature matrix, so as to perform attention weighting on the first feature matrix and obtain the second feature matrix;

[0202] Step S430: Decompose the second feature matrix into trend components and periodic components;

[0203] Step S440: Perform frequency domain attention processing on the periodic component to obtain the first component;

[0204] Step S450: Combine the first component and the trend component to obtain the third feature matrix;

[0205] Step S460: Predict the third feature matrix using the normalized exponential function to obtain the prediction result.

[0206] In this embodiment, the multi-dimensional multimodal feature matrix is ​​first projected from the time domain to the frequency domain through linear transformation. Based on the frequency domain sparsity of physiological signals, effective high-frequency and low-frequency components in the signal are selected and retained, while noise interference and invalid redundant feature information in the frequency domain are eliminated. Then, the selected frequency domain features are dimensionally completed and calibrated. Finally, the optimized frequency domain features are remapped back to the time domain through inverse transformation, resulting in a first feature matrix with regular dimensions and high feature purity. This retains the core effective information of the multimodal features while significantly reducing the overall complexity of feature operations, and enhances the feature representation capability of non-stationary physiological time-series signals such as EEG, ECG, and brain oxygenation.

[0207] Furthermore, the first feature matrix is ​​decomposed into three types of feature matrices according to feature dimensions: a query matrix, a key matrix, and a value matrix, all of which are dimensionally perfectly matched. Then, in the frequency domain, the vector correlation between the three is calculated. This correlation value directly reflects the importance of each type of feature in predicting severe illness. Based on the calculated correlation values, corresponding attention weights are generated, assigning high weights to key, effective features and low weights to worthless, noisy features. These weights are then used to weight and integrate the value matrix, ultimately resulting in the attention-weighted second feature matrix. This allows the model to accurately focus on feature information with core value in judging severe illness while weakening ineffective, interfering features.

[0208] Furthermore, the weighted and optimized feature matrix is ​​further refined by decomposing the complex second feature matrix into two independent, complementary, and non-overlapping feature components. This enables targeted and differentiated processing of different feature patterns, avoids pattern confusion caused by mixed feature modeling, and significantly improves the accuracy of subsequent feature processing.

[0209] Specifically, the trend component represents the slow-changing characteristics of the feature matrix, corresponding to the long-term development pattern of the critically ill patient's condition. For example, the overall improvement or steady deterioration of the patient's physiological indicators is a basic characteristic reflecting the core trend of the condition. This component retains the original characteristic pattern throughout without modification. The periodic component represents the rhythmic fluctuation characteristics of the feature matrix, corresponding to the periodic peaks and troughs of the EEG signal, the heart rate cycle of the ECG signal, the metabolic cycle of the brain oxygen signal, and other physiological rhythm changes. These periodic fluctuations are key characteristics for judging the patient's state of consciousness and sudden changes in the condition.

[0210] Furthermore, the vector correlation of the feature matrix of the periodic component is recalculated in the frequency domain to generate exclusive attention weights adapted to the periodic features, thereby further strengthening the abnormal rhythmic features in the periodic component that are highly correlated with the disease condition, while suppressing invalid noise fluctuations in the periodic component. Then, the weighted periodic features are integrated in dimensions and restored in features to finally form the optimized periodic features, namely the first component, in order to maximize the clinical reference value of the periodic components, make the representation of the periodic features more accurate and the regularity clearer, and effectively amplify the rhythmic feature signals that are meaningful for disease prediction.

[0211] Furthermore, the optimized periodic features are organically integrated with the original trend features to restore the complete feature patterns. During the combination process, the optimized first component and the unmodified trend component are precisely aligned according to the feature dimensions to complete feature splicing and fusion. This fusion is based on the complementary integration of feature dimensions, ensuring that the reconstructed feature matrix includes both the long-term development trend of the patient's condition and the double-optimized physiological rhythm fluctuation features. The resulting third feature matrix is ​​a complete feature set that takes into account both the global trend and key local periodic features. It retains the core direction of disease development while encompassing precisely optimized rhythmic details. The feature matrix has no missing information and no redundant superposition, completely restoring all the core feature patterns after the combination of physiological signals and clinical information of critically ill patients, providing complete and accurate feature input for the final disease prediction.

[0212] Furthermore, the third feature matrix is ​​first compressed to a reasonable degree to extract core high-dimensional feature vectors that directly represent the disease state. These feature vectors are then input into a normalized exponential function to perform probabilistic calculations. The normalized exponential function maps the numerical results of the feature vectors to a range of zero to one. Each output value corresponds to the probability of different disease states in critically ill patients, and the sum of the probabilities of all disease states is one. Based on this probability distribution, the category with the highest probability value is selected as the final prediction result.

[0213] Specifically, the prediction result is the category of the patient's state of consciousness or disease progression, such as conscious, comatose, or critically ill. At the same time, the probability value corresponding to each category can be output simultaneously, which not only meets the clinical diagnosis and treatment needs for clear results, but also provides a quantitative probability reference.

[0214] In some embodiments, the frequency enhancement prediction model in step S150 is trained through the following steps S510 to S540:

[0215] Step S510: Obtain training data, which includes a multimodal feature matrix and is associated with the actual disease status category label of the critically ill patients.

[0216] Step S520: Construct the initial training model based on the frequency domain enhancement module, frequency domain attention module, hybrid expert decomposition module, cross-modal interaction unit, and two-dimensional periodic transformation and multi-scale convolution unit;

[0217] Step S530: Train the initial training model based on the training data to obtain the frequency enhancement prediction model.

[0218] In this embodiment, training data is first acquired, which includes a multimodal feature matrix and is associated with the actual disease status category label corresponding to critically ill patients. Preferably, frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients are extracted from the EEG, ECG, and cerebral oxygen multimodal time-series signals collected by sensors. Combined with static features extracted from medical record data, feature enhancement is performed through random convolution kernel transformation. Then, a complete feature matrix is ​​obtained through intramodal fusion, cross-modal fusion, and static feature fusion. This feature matrix combines the dynamic time-frequency features of multimodal physiological signals with the static features of the patient's clinical background, and has complete feature dimensions without redundant information. Meanwhile, the training data has supervised labeling attributes. Each multimodal feature matrix corresponds one-to-one with the actual disease status category of critically ill patients during the same period. This labeling is the actual result determined by clinical medical staff according to the diagnosis and treatment guidelines, including disease status categories such as conscious, different degrees of coma, and critical condition. The labeling information is real and accurate, providing a clear fitting basis for the subsequent supervised training of the model. Before actual training, the training data will also be standardized, including dividing the training subset and validation subset, and performing data balancing and adaptation to ensure the effectiveness and rationality of the training data, laying a solid data foundation for the accuracy and generalization ability of the model training.

[0219] Furthermore, based on the technical logic of severe illness prediction, an organically linked, functionally independent, and collaboratively operating end-to-end complete model architecture is formed. All modules and units are core components of this model, none of which can be omitted and they are interconnected and adapted to each other, jointly serving the frequency domain enhancement processing and accurate prediction of multimodal feature matrices.

[0220] The model comprises several key modules: a frequency domain enhancement module, a core module at the beginning, which projects the input multimodal feature matrix from the time domain to the frequency domain, filters effective frequency components, and reduces computational complexity; a frequency domain attention module, which receives the output of the frequency domain enhancement module and performs attention weighting on features in the frequency domain, focusing on key features with core value for disease prediction; a hybrid expert decomposition module, which decomposes the weighted feature matrix into trend and periodic components for refined processing; a cross-modal interaction unit, which specifically mines the coupling relationships between features of different modalities and optimizes the contribution of each modal feature by generating modal attention weights; and a two-dimensional periodic transformation and multi-scale convolution unit, which captures multi-scale periodic patterns in the feature matrix and strengthens the rhythmic feature expression of physiological signals. The modules and units are sequentially connected according to signal processing logic, and the model's end is adapted to a normalized exponential function output layer for multi-class prediction. The overall architecture fully covers the entire process from feature input to prediction result output without functional gaps. All modules of this initial training model have pre-set trainable initial weight parameters, enabling direct iterative training.

[0221] Specifically, the predefined training subset is input into the initial training model. Through collaborative computation of its internal modules, the initial training model performs a full-process processing on the input multimodal feature matrix, including frequency domain enhancement, attention weighting, component decomposition, cross-modal interaction, and periodic feature extraction, ultimately outputting the predicted probability of the corresponding severe illness state category. Then, the loss difference between the prediction result output by the initial training model and the actual illness state category labels associated with the training data is calculated. This difference is quantified using a loss function adapted to multi-classification tasks, and then fed back to each module and unit of the initial training model via backpropagation, updating and optimizing all pre-set trainable weight parameters sequentially.

[0222] Throughout the iterative training process, the validation subset is input into the model for real-time performance validation. Regularization and early stopping strategies are used to prevent overfitting, and an adaptive optimization strategy is employed to adjust the learning rate, allowing the model's weights to continuously converge towards the optimal direction. Specifically, training terminates when the model's core performance metrics, such as prediction accuracy and stability on the validation subset, stabilize and no longer improve. The model at this point, with all parameters optimized, is considered a mature frequency augmentation prediction model.

[0223] In some embodiments, in step S530, the initial training model is trained based on the training data to obtain a frequency enhancement prediction model, including the following steps S610 to S620:

[0224] Step S610: Input the training data into the initial training model to obtain the initial prediction results output by the initial training model;

[0225] Step S620: Using the cross-entropy loss function, train the initial training model based on the training data and the initial prediction results to obtain the frequency enhancement prediction model.

[0226] In this embodiment, after the training data is input into the initial training model, it passes through each core module in sequence according to the model's preset technical process: first, the frequency domain enhancement module completes the projection from the time domain to the frequency domain and the effective component selection; then, the frequency domain attention module strengthens the weights of key features; the hybrid expert decomposition module splits and processes trend and periodic components; the cross-modal interaction unit explores the coupling relationship between modes; the two-dimensional periodic transformation and multi-scale convolution unit capture multi-scale periodic patterns; finally, the normalized exponential function layer at the end of the model completes the calculation and outputs the initial prediction result. This initial prediction result is in the form of a probability distribution, corresponding to the probability of occurrence of various disease states (such as conscious, comatose, and death) of critically ill patients, and the sum of the probabilities of all categories is 1. It directly reflects the prediction ability of the initial training model in the unoptimized state and is the core basis for subsequent loss calculation and model parameter adjustment.

[0227] Furthermore, the prediction error is quantified using the cross-entropy loss function, and the model parameters are continuously adjusted using the backpropagation mechanism. Ultimately, the initial trained model is optimized into a frequency-enhanced prediction model with stable performance and high accuracy, perfectly aligning with the characteristics of multi-classification tasks and the actual needs of severe illness prediction. Specifically, the cross-entropy loss function is used to accurately calculate the difference between the initial prediction results and the actual disease status category labels in the training data. The actual labels participate in the calculation using one-hot encoding, and the magnitude of the loss value directly reflects the degree of deviation between the model's prediction results and the actual situation; the smaller the loss value, the more accurate the prediction.

[0228] Furthermore, after calculating the loss value, the model initiates a backpropagation mechanism, transmitting the loss signal backward along the model architecture to all core modules (frequency domain enhancement module, frequency domain attention module, etc.) and units. This updates the pre-set trainable weight parameters and bias parameters within each module round by round, always in the direction of gradient descent that reduces the loss value. Multiple optimization strategies are also implemented during training.

[0229] For example, regularization and early stopping strategies are combined to suppress overfitting and prevent the model from overfitting the training data and losing its generalization ability; an adaptive optimizer and cosine annealing learning rate scheduling strategy are used to balance training speed and convergence stability; gradient clipping is used to control the gradient norm to prevent gradient explosion from causing training failure; at the same time, cross-validation is used to make full use of the sample data, and class weights are added to the loss function to alleviate the impact of data imbalance.

[0230] Furthermore, during the training process, the model iteratively executes the process of "input training data - output prediction results - calculate loss - backpropagation to update parameters" in batches. Simultaneously, it monitors core performance indicators such as prediction accuracy and F1 score in real time through a validation subset. When the indicators tend to stabilize for several consecutive rounds and no longer improve significantly, the training is terminated. At this point, the parameters of all core modules have been optimized to the optimal state, and the initial training model is formally transformed into a mature frequency-enhanced prediction model that can be directly applied to clinical critical care prediction scenarios.

[0231] In some embodiments, after inputting the multimodal feature matrix into the frequency enhancement prediction model in step S160 and obtaining the prediction result output by the frequency enhancement prediction model, the method further includes the following steps S710 to S720:

[0232] Step S710: Determine the user's state of consciousness based on the prediction results;

[0233] Step S720: Generate a corresponding monitoring strategy based on the state of consciousness.

[0234] In this embodiment, the model outputs a multi-class probability distribution, where each probability value corresponds to a different category of consciousness state in critically ill patients (e.g., conscious, mild coma, deep coma, death, etc.), and the sum of the probabilities of all categories is 1. During the judgment process, the category with the highest probability value is prioritized as the core judgment criterion. Simultaneously, the confidence threshold corresponding to model validation metrics (e.g., accuracy, recall) is referenced. If the highest probability value does not reach the preset confidence threshold, the next highest probability category and the core features of the patient's multimodal physiological signals (e.g., frequency domain energy distribution of EEG signals, rhythm stability of ECG signals) are combined for auxiliary judgment to avoid misjudgment caused by a single probability value. Therefore, the final determined state of consciousness must be consistent with the labeled categories in the training data and conform to the standardized classification definition of consciousness states in clinical practice for critically ill patients, ensuring that the judgment results have clinical acceptance and operability, and providing a clear decision-making basis for subsequent development of targeted monitoring strategies.

[0235] Furthermore, if the user is determined to be conscious, the monitoring strategy primarily involves routine dynamic monitoring, with a clearly defined monitoring frequency (e.g., recording core physiological indicators hourly). Emphasis is placed on changes in the rhythm of electrocardiogram (ECG) signals and the stability of brain oxygen saturation. Simultaneously, based on underlying disease information from the medical record, a routine nursing care and condition observation plan is developed, requiring no additional intensive intervention. If the user is in a coma (including mild and deep coma), the core strategy is to enhance brain function and vital sign monitoring, increasing the sampling and analysis frequency of EEG signals. The focus is on tracking abnormal fluctuations in the EEG frequency domain amplitude spectrum and local abnormal activity reflected by multi-scale wavelet coefficients. Simultaneously, the monitoring intervals for ECG and brain oxygen signals are increased, and the necessary interventions are clearly defined. Emergency intervention threshold indicators (such as initiating intervention when cerebral oxygen saturation falls below a specific range) are established, and targeted nursing measures (such as postural adjustment and complication prevention) are developed in conjunction with the patient's medication and surgical history in the medical record. If the patient is determined to be in a state with extremely high risk of death, the strategy focuses on end-of-life care and emergency preparedness, clarifying the continuous monitoring plan for core vital signs, linking with clinical emergency procedures, and developing a rapid response mechanism for abnormal multimodal signals. This ensures seamless integration between the monitoring strategy and clinical emergency procedures, thereby achieving both accurate consciousness prediction based on the model and full correlation with the core characteristics of multimodal physiological signals and the patient's clinical background information, adapting to the actual clinical needs of intensive care scenarios.

[0236] like Figure 7 As shown in some embodiments of this application, a multimodal frequency domain enhanced critical illness prediction device is provided. The device includes an acquisition module 710, an extraction module 720, a feature module 730, a fusion module 740, a construction module 750, and a prediction module 760. Specifically:

[0237] The acquisition module 710 is used to acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by the sensor within the same time range; the second data includes the user's medical record data.

[0238] The extraction module 720 is used to extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal.

[0239] Feature module 730 is used to extract static features from medical record data;

[0240] The fusion module 740 is used to fuse static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time-series signal to obtain a multimodal feature matrix;

[0241] Module 750 is used to build a frequency-enhanced prediction model; the frequency-enhanced prediction model is used to enhance the input of the frequency-enhanced prediction model in the frequency domain to predict the corresponding state probability.

[0242] The prediction module 760 is used to input the multimodal feature matrix into the frequency enhancement prediction model and obtain the prediction results output by the frequency enhancement prediction model.

[0243] It should be noted that the multimodal frequency domain enhanced critical illness prediction device provided in this embodiment is based on the same inventive concept as the multimodal frequency domain enhanced critical illness prediction method described above. Therefore, the relevant content of the multimodal frequency domain enhanced critical illness prediction method described above also applies to the content of the multimodal frequency domain enhanced critical illness prediction device. Therefore, it will not be repeated here.

[0244] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal frequency domain enhanced critical illness prediction method.

[0245] like Figure 8 , Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0246] At least one battery;

[0247] At least one memory;

[0248] At least one processor;

[0249] At least one program;

[0250] The program is stored in memory, and the processor executes at least one program to implement the multimodal frequency domain enhanced critical illness prediction method described above in this disclosure.

[0251] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0252] The electronic devices according to embodiments of this application will now be described in detail.

[0253] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0254] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement a multimodal frequency domain enhanced critical illness prediction method according to an embodiment of this disclosure.

[0255] The input / output interface 1800 is used to implement information input and output.

[0256] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0257] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0258] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0259] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described multimodal frequency domain enhanced critical illness prediction method.

[0260] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0261] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0262] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0263] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0264] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0265] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any related variations, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0266] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0267] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0268] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0269] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0270] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0271] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

[0272] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A multi-modal frequency domain severe case prediction method, characterized in that, The method includes: Acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by sensors within the same time range; the second data includes the user's medical record data; Extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal; Extract static features from the medical record data; By integrating the static features and the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal, a multimodal feature matrix is ​​obtained; A frequency-enhanced prediction model is constructed; the frequency-enhanced prediction model is used to enhance the input of the frequency-enhanced prediction model in the frequency domain to predict the corresponding state probability; The multimodal feature matrix is ​​input into the frequency enhancement prediction model to obtain the prediction result output by the frequency enhancement prediction model; The multiple modal time-series signals include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and brain oxygenation signals. The extraction of the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal includes: The electroencephalogram (EEG) signal, the electrocardiogram (ECG) signal, and the brain oxygenation signal are respectively segmented to obtain a first signal, a second signal, and a third signal; Perform Discrete Fourier Transform on the first signal, the second signal, and the third signal respectively to obtain the frequency domain amplitude spectrum, the phase spectrum of the first signal, the frequency domain amplitude spectrum, the phase spectrum of the second signal, the frequency domain amplitude spectrum, and the phase spectrum of the third signal; Perform wavelet transform on the first signal, the second signal and the third signal respectively to obtain the multi-scale wavelet coefficients of the first signal, the multi-scale wavelet coefficients of the second signal and the multi-scale wavelet coefficients of the third signal; The frequency enhancement prediction model outputs prediction results through the following steps: The multimodal feature matrix is ​​enhanced in the frequency domain to obtain the first feature matrix; Calculate the vector correlations corresponding to the query matrix, key matrix, and value matrix of the first feature matrix in the frequency domain of the first feature matrix, so as to perform attention weighting on the first feature matrix to obtain the second feature matrix; The second characteristic matrix is ​​decomposed into trend components and periodic components; The first component is obtained by performing frequency domain attention processing on the periodic component. The first component and the trend component are combined to obtain the third feature matrix; The prediction result is obtained by predicting the third feature matrix using a normalized exponential function.

2. The multi-modal frequency domain severity prediction method of claim 1, wherein, The multimodal feature matrix is ​​obtained by fusing the static features and the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal, including: The static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time series signal are respectively subjected to random convolution kernel transformation to obtain the enhanced static features, the enhanced frequency domain amplitude spectrum, the enhanced phase spectrum and the enhanced multi-scale wavelet coefficients corresponding to each modal time series signal; The enhanced frequency domain amplitude spectrum, enhanced phase spectrum, and enhanced multi-scale wavelet coefficients corresponding to each modal time series signal are fused to obtain the multimodal characteristic vector corresponding to each modal time series signal. The enhanced static features and the multimodal characteristic vectors corresponding to each modal time-series signal are fused to obtain the multimodal feature matrix.

3. The multi-modal frequency domain severity prediction method of claim 1, wherein, The frequency enhancement prediction model is trained through the following steps: Acquire training data, which includes a multimodal feature matrix and is associated with the actual disease status category label of the critically ill patient. An initial training model is constructed based on a frequency domain enhancement module, a frequency domain attention module, a hybrid expert decomposition module, a cross-modal interaction unit, and a two-dimensional periodic transformation and multi-scale convolution unit. The initial training model is trained based on the training data to obtain the frequency enhancement prediction model.

4. The multi-modal frequency domain augmented severe case prediction method of claim 3, wherein, The step of training the initial training model based on the training data to obtain the frequency enhancement prediction model includes: The training data is input into the initial training model to obtain the initial prediction result output by the initial training model; The frequency enhancement prediction model is obtained by training the initial training model using the cross-entropy loss function based on the training data and the initial prediction results.

5. The multimodal frequency domain enhanced critical illness prediction method according to claim 1, characterized in that, After inputting the multimodal feature matrix into the frequency enhancement prediction model and obtaining the prediction result output by the frequency enhancement prediction model, the method further includes: Based on the prediction results, the user's state of consciousness is determined; Based on the stated state of consciousness, a corresponding monitoring strategy is generated.

6. A multimodal frequency domain enhanced critical illness prediction device, characterized in that, The device includes: The acquisition module is used to acquire first data and second data; the first data includes multiple modal time-series signals of the user collected by sensors within the same time range; the second data includes the user's medical record data. The extraction module is used to extract the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal; The feature module is used to extract static features from the medical record data; The fusion module is used to fuse the static features and the frequency domain amplitude spectrum, phase spectrum and multi-scale wavelet coefficients corresponding to each modal time-series signal to obtain a multimodal feature matrix; A construction module is used to construct a frequency-enhanced prediction model; the frequency-enhanced prediction model is used to enhance the input of the frequency-enhanced prediction model in the frequency domain to predict the corresponding state probability; The prediction module is used to input the multimodal feature matrix into the frequency enhancement prediction model to obtain the prediction result output by the frequency enhancement prediction model; The multiple modal time-series signals include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and brain oxygenation signals. The extraction of the frequency domain amplitude spectrum, phase spectrum, and multi-scale wavelet coefficients corresponding to each modal time-series signal includes: The electroencephalogram (EEG) signal, the electrocardiogram (ECG) signal, and the brain oxygenation signal are respectively segmented to obtain a first signal, a second signal, and a third signal; Perform Discrete Fourier Transform on the first signal, the second signal, and the third signal respectively to obtain the frequency domain amplitude spectrum, the phase spectrum of the first signal, the frequency domain amplitude spectrum, the phase spectrum of the second signal, the frequency domain amplitude spectrum, and the phase spectrum of the third signal; Perform wavelet transform on the first signal, the second signal and the third signal respectively to obtain the multi-scale wavelet coefficients of the first signal, the multi-scale wavelet coefficients of the second signal and the multi-scale wavelet coefficients of the third signal; The frequency enhancement prediction model outputs prediction results through the following steps: The multimodal feature matrix is ​​enhanced in the frequency domain to obtain the first feature matrix; Calculate the vector correlations corresponding to the query matrix, key matrix, and value matrix of the first feature matrix in the frequency domain of the first feature matrix, so as to perform attention weighting on the first feature matrix to obtain the second feature matrix; The second characteristic matrix is ​​decomposed into trend components and periodic components; The first component is obtained by performing frequency domain attention processing on the periodic component. The first component and the trend component are combined to obtain the third feature matrix; The prediction result is obtained by predicting the third feature matrix using a normalized exponential function.

7. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform a multimodal frequency domain enhanced critical illness prediction method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the multimodal frequency domain enhanced critical illness prediction method according to any one of claims 1 to 5.

Images