Multi-modal physiological electric signal processing method based on deep learning

Abstract

The invention provides a multi-modal physiological electric signal processing method based on deep learning, and relates to the technical field of signal processing. The method comprises the following steps: acquiring and preprocessing a multi-modal physiological electric signal original sequence, and constructing a multi-modal physiological electric signal data set; constructing a cross-modal time delay distribution response structure based on the multi-modal physiological electric signal data set, and introducing an intrinsic time delay relationship between modals to form a cross-modal time delay collaborative embedding sequence; further calculating the disturbance quantity of the multi-channel physiological electric signals, introducing a disturbance coherence factor, forming a dynamic coherence modulation sequence through time recursion, and performing channel-level modulation on the cross-modal time delay collaborative embedding sequence; then, a multi-scale differential structure is constructed, a state hopping coding sequence is generated, and a multi-modal structure expression sequence is obtained; and constructing a deep learning signal processing model based on the multi-modal structure expression sequence, forming a feature expression sequence for signal generation, executing decoding reconstruction processing, and outputting a multi-modal physiological electric signal processing result.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to a multimodal physiological electrical signal processing method based on deep learning. Background Technology

[0002] Currently, physiological electrical signals, as an important information carrier reflecting changes in human physiological state, are widely used in clinical monitoring, health assessment, and intelligent medical technology. Especially in application scenarios where multiple physiological modalities such as electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG) are simultaneously acquired and analyzed, multimodal physiological electrical signal processing methods are of great significance for improving the accuracy and stability of state recognition. With the continuous development of wearable devices and multi-channel acquisition systems, physiological electrical signals exhibit diverse modalities, a large number of channels, and complex temporal variations during acquisition. There are significant differences between different physiological modalities in response timing, amplitude changes, and structural characteristics. There is an urgent need to establish processing methods that can effectively characterize the overall features of multimodal physiological electrical signals to meet the application needs of complex physiological state analysis.

[0003] Publication No. CN110742349A discloses a physiological signal analysis method based on the fusion of time-domain and frequency-domain features. This method extracts statistical features from multi-channel physiological electrical signals and constructs a classification model to identify physiological states. Publication No. CN113197579A discloses a physiological signal analysis method based on multimodal information fusion. This method extracts and fuses features from physiological electrical signals from different physiological modalities, and combines them with a deep learning model to analyze and identify human physiological states.

[0004] However, existing technologies mostly focus on modeling the local features and single time-scale features of physiological electrical signals, failing to fully consider the overall correlation between multimodal physiological electrical signals at the time response and channel structure levels. They also lack a systematic modeling method for the continuity of structural changes in the signal over time. In application scenarios where multimodal collaborative changes are significant and signal states switch rapidly, problems such as unstable feature representation, insufficient structural consistency, and insufficient continuity of processing results arise, thus limiting the applicability of existing methods in complex physiological electrical signal processing tasks. Summary of the Invention

[0005] This invention proposes a deep learning-based method for processing multimodal physiological electrical signals. Addressing the challenges of significant differences in modal response timing, complex inter-channel structural relationships, and difficulty in characterizing the continuity of signal state changes during joint acquisition and analysis of multimodal physiological electrical signals, this method constructs a multi-stage signal processing mechanism integrating cross-modal time delay modeling, perturbation coherent modulation, and multi-scale jump coding. First, the method acquires the raw sequence data of multimodal physiological electrical signals and constructs a multimodal physiological electrical signal dataset. Based on the intrinsic time delay relationships of each physiological mode, a cross-modal time delay distributed response structure is constructed to form a cross-modal time delay co-embedding sequence. Furthermore, by calculating the perturbation amount of the multi-channel physiological electrical signals and introducing a perturbation coherence factor combined with a time recursion mechanism, a dynamic coherent modulation sequence is constructed to address the cross-modal time delay co-embedding. The method performs channel-level modulation processing on the embedded sequence, and further constructs a multi-scale differential structure based on the modulated cross-modal feature sequence. It models the signal change amplitude at different time scales and forms a state transition coding sequence to obtain a multimodal structural expression sequence for signal processing. Then, a deep learning signal processing model is constructed using the multimodal structural expression sequence as input. Through feature expression and structure reconstruction training, feature representation sequences reflecting the temporal structure and channel correlation characteristics of multimodal physiological electrical signals are extracted. Finally, the feature representation sequences are decoded to generate multimodal physiological electrical signal processing results, which are used as the output of this method. This achieves unified modeling and stable output of the structural characteristics and state change process of multimodal physiological electrical signals, and improves the structural expression ability and state recognition stability of multimodal physiological electrical signals.

[0006] A deep learning-based method for processing multimodal physiological electrical signals, the specific method is as follows: S1. Obtain the raw sequence data of multimodal physiological electrical signals, preprocess the raw sequence data, and construct a multimodal physiological electrical signal dataset; S2. Based on the multimodal physiological electrical signal dataset, construct a cross-modal time-delay distributed response structure, introduce the intrinsic time-delay relationship between modes, and form a cross-modal time-delay cooperative embedding sequence; S3. Based on the cross-modal time-delay collaborative embedding sequence, calculate the perturbation amount of the multi-channel physiological electrical signal, introduce the perturbation coherence factor, and form a dynamic coherent modulation sequence through time recursion to perform channel-level modulation processing on the cross-modal time-delay collaborative embedding sequence. S4. Based on the modulated cross-modal time-delay co-embedding sequence, construct a multi-scale differential structure for multi-channel physiological electrical signals and generate a state transition coding sequence to characterize the channel change characteristics, thereby obtaining a multi-modal structure expression sequence for signal processing. S5. Based on the multimodal structural expression sequence, construct a deep learning signal processing model, perform feature expression and structural reconstruction training, and form a feature representation sequence for signal generation; S6. Based on the deep learning signal processing model, perform decoding and reconstruction processing on the feature representation sequence formed in steps S2 to S5, and output the reconstruction result as the result of multimodal physiological electrical signal processing.

[0007] Preferably, for the construction of a multimodal physiological electrical signal dataset, the raw physiological electrical signal sequences under different physiological modalities are first acquired using a human multimodal physiological electrical signal acquisition system. The physiological modalities include electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG) modalities. The physiological electrical signals under each physiological modality are synchronously acquired by the corresponding multi-channel sensors. During the signal acquisition phase, the sampling rate parameter is uniformly set for all physiological electrical signal channels and remains unchanged throughout the acquisition process. Simultaneously, the modality identification information, signal channel number information, and time index information corresponding to each sampling moment are recorded to ensure strict alignment of different physiological modalities and different signal channels in the time dimension. After the raw physiological electrical signal acquisition is completed, preprocessing operations are performed on the physiological electrical signal sequences under each modality. Preprocessing includes standardizing the signal data format. The physiological electrical signals were processed by filtering them with a digital filter with a preset passband range to remove high-frequency noise and low-frequency baseline drift introduced during acquisition. Amplitude normalization was then performed on the filtered physiological electrical signal sequences to unify the amplitude scale of physiological electrical signals under different physiological modalities and signal channels. Amplitude limiting and smoothing were then applied to abrupt sampling points in the signals to enhance the continuity of the physiological electrical signals over time. After the above preprocessing, the physiological electrical signal sequences under each physiological modality were time-aligned and organized according to a fixed time step. The preprocessed physiological electrical signal sequences were then uniformly organized with the corresponding modality identification information, signal channel number information, and time index information to construct a multimodal physiological electrical signal dataset containing a multimodal, multi-channel time series structure.

[0008] Preferably, multimodal physiological electrical signals exhibit significant non-stationary characteristics in the time dimension, and the signal structures of different physiological modes have differences in amplitude variation and response delay. The original sequence of a single mode cannot fully express the cross-modal cooperative structure. Based on the channel structure arrangement and sequence alignment mechanism, this invention constructs multimodal physiological electrical signals into a cross-modal continuous structure with a unified time resolution. While maintaining the integrity of the signal structure identity, it solidifies the correspondence between modality identification, channel number and time index, providing a continuous and stable data input foundation for the construction of cross-modal time delay distribution response, dynamic perturbation calculation and multi-scale differential structure modeling.

[0009] Furthermore, in step S2, the construction process of the cross-modal time delay distribution response structure includes: based on the multimodal physiological electrical signal dataset, for each physiological mode, extracting multiple sets of modality-related intrinsic time delays within a preset time range for the physiological electrical signal sequence, and forming the time delay distribution response corresponding to the physiological mode based on the time offset relationship corresponding to different intrinsic time delays; Based on the time delay distribution response of each physiological mode, and based on the intrinsic time delay of each physiological mode, the time delay correlation between physiological modes is introduced, and the time delay distribution response under different physiological modes is synergistically combined to obtain the cross-modal time delay distribution response structure. Based on the cross-modal time-delay distribution response structure, multimodal physiological electrical signal sequences are weighted and combined to form cross-modal time-delay collaborative embedding sequences.

[0010] Furthermore, addressing the issue of significant time delay differences and inconsistent temporal structures among modalities in multimodal physiological electrical signals, this invention proposes a modeling mechanism combining cross-modal time delay distribution construction and sequence collaborative embedding. First, based on the multimodal physiological electrical signal dataset, the sequence structures of different physiological modalities are organized under a unified time index to form a modal sequence structure describing the positional relationship of signal sampling times. Then, within a fixed time range, the intrinsic time delay difference between different physiological modalities is calculated based on the modal sequence structure, and a cross-modal time delay mapping is established based on the signal distribution relationship on the time index, forming a cross-modal time delay distribution response structure. This structure can express the continuity and synergy of cross-modal time differences and solidify the time delay relationship. Based on this, the cross-modal time delay mapping relationship is applied to the time axes of different physiological modalities. The time axis differences are normalized through joint alignment, and the corresponding times of each modality are connected into a unified structure sequence, forming a cross-modal time delay collaborative embedding sequence. This embedding sequence can achieve cross-modal structure fusion while maintaining the integrity of the temporal identity of each physiological modality, providing a stable data input foundation for subsequent channel-level perturbation calculation and dynamic coherent modulation.

[0011] Preferably, multimodal physiological electrical signals exhibit significant intrinsic time delay differences and inconsistent response structures between modes. This invention strengthens the ability to express multimodal temporal relationships and improves the stability and structural consistency of cross-modal continuous modeling by constructing cross-modal time delay distribution response structures and forming cross-modal time delay collaborative embedding sequences.

[0012] Furthermore, in step S3, based on the cross-modal time-delay co-embedding sequence, the signal difference of each physiological electrical signal channel at adjacent time points is calculated along the time dimension to obtain the perturbation sequence of the corresponding channel; Based on the perturbation sequence of each channel, the perturbation difference between any pair of channels is calculated, and a perturbation coherence factor is introduced according to the perturbation difference to weight and combine the perturbation responses of multiple channels to form a channel-level perturbation coherence response sequence. Time recursion processing is performed on the channel-level perturbation coherent response sequence to construct a dynamic coherent modulation sequence that evolves over time. The dynamic coherent modulation sequence is then used to perform channel-level modulation on the cross-modal time-delay co-embedding sequence to obtain the perturbation-coherently modulated cross-modal feature sequence.

[0013] Furthermore, addressing the issues of significant differences in perturbation distribution, uneven structural fluctuation intensity, and complex continuity of temporal response changes in multimodal physiological electrical signals across channels, this invention, after obtaining the perturbation-coherently modulated cross-modal feature sequence, first constructs a channel-level perturbation difference expression structure based on the multi-channel perturbation quantity sequence to describe the joint change relationship of perturbation responses between channels in the time dimension, improving the consistency of channel feature expression in the collaborative modeling process; subsequently, based on the channel-level perturbation difference relationship, a perturbation coherence factor weighted combination processing is performed, and a continuous perturbation response is constructed along the time index. The distributed structure can enhance cross-channel structural correlation while maintaining the integrity of the perturbation trend. Based on this, the dynamic coherent modulation sequence is recursively pushed along the time dimension, so that the perturbation diffusion forms a directional propagation mode in the sequence structure, thereby constructing a time-varying perturbation coherent modulation field, so that the multi-channel perturbation structure has stable and continuous structural connection characteristics in the global expression. Finally, the perturbation coherent modulation field and cross-modal time delay are co-embedded in the sequence for structural fusion processing to form a perturbation-extended and enhanced cross-modal feature sequence, which provides a continuous and stable structural input foundation for subsequent multi-scale differential structure modeling.

[0014] Preferably, multimodal physiological electrical signals exhibit significant perturbation differences, inconsistent response change directions, and uneven channel coupling between different channels. By constructing a perturbation quantity sequence and introducing a perturbation coherence factor, the cross-channel perturbation relationship can be effectively characterized, enabling the perturbation expression structure to maintain continuity and stability in the time dimension. Combined with a time recursion mechanism, a dynamic coherent modulation sequence is formed, which helps to enhance the channel consistency expression capability in the cross-modal feature fusion process and improve the structural sensitivity and state recognition accuracy in the subsequent multi-scale differential structure modeling stage.

[0015] Furthermore, in step S4, based on the perturbation-coherently modulated cross-modal feature sequence, a multi-scale differential structure is constructed for each physiological electrical signal channel along the time dimension, and the signal change amplitude between adjacent sampling points is calculated at different time scales to form a multi-scale differential response sequence; Based on the multi-scale differential response sequence, the differential responses at each time scale are weighted and fused to obtain a scale fusion jump intensity sequence that characterizes the intensity of changes in the state of physiological electrical signals. Based on the scale fusion jump intensity sequence, the state changes of the physiological electrical signal channel are discriminated at each time index to form the corresponding state jump coding sequence. The state jump coding sequence is determined by the numerical relationship between the scale fusion jump intensity and the preset judgment threshold, and different discrete coding states are taken under different numerical relationships. These are organized as discrete state components in the multimodal structure expression sequence to obtain the multimodal structure expression sequence for subsequent signal processing.

[0016] Furthermore, addressing the issue of significant differences in state change scales and uneven channel response intensity exhibited by the perturbed coherent modulation cross-modal feature sequence during time evolution, this invention, after acquiring the cross-modal feature sequence, constructs a multi-scale differential structure for each physiological electrical signal channel along the time dimension, and calculates the signal change amplitude between adjacent sampling points at different time scales, forming a multi-scale differential response sequence. This decomposes the channel change behavior in the continuous time series into differential response forms with clear time scale affiliations. Subsequently, based on the multi-scale differential response sequence, weighted fusion processing is performed on the differential responses at each time scale to obtain a representation of the physiological electrical signal. The scale-fused jump intensity sequence of signal state change intensity is used to uniformly characterize the contribution relationship of differential responses at different time scales to the overall state change. Based on this, the state changes of the physiological electrical signal channel are discriminated at each time index according to the scale-fused jump intensity sequence, forming a corresponding state jump encoding sequence. The state jump encoding sequence is determined by the numerical relationship between the scale-fused jump intensity and the preset judgment threshold, and different discrete encoding states are taken under different numerical relationships. Finally, the state jump encoding sequence is organized as a discrete state component in the multimodal structure expression sequence to obtain a multimodal structure expression sequence for subsequent signal processing.

[0017] Preferably, the cross-modal feature sequence after perturbation coherent modulation exhibits inconsistent signal variation amplitude distribution at different time scales in the time dimension. To address this characteristic, this invention constructs a multi-scale differential structure modeling mechanism, building a multi-scale differential structure for each physiological electrical signal channel along the time dimension, and calculating the signal variation amplitude between adjacent sampling points at different time scales to form a multi-scale differential response sequence. Further, based on the multi-scale differential response sequence, the differential responses at each time scale are weighted and fused to obtain a scale-fused jump intensity sequence characterizing the intensity of physiological electrical signal state changes. On this basis, according to the scale-fused jump intensity sequence, the state changes of the physiological electrical signal channel are discriminated at each time index to form a corresponding state jump encoding sequence. The state jump encoding sequence is determined by the numerical relationship between the scale-fused jump intensity and a preset judgment threshold, and different discrete encoding states are taken under different numerical relationships. Finally, the state jump encoding sequence is organized as a discrete state component in the multi-modal structure expression sequence to obtain a multi-modal structure expression sequence for subsequent signal processing.

[0018] Furthermore, in step S5, a deep learning signal processing model is constructed based on the fused multimodal structural expression sequence. The multimodal structural expression sequence is used as the model input to establish a feature mapping relationship for characterizing the structural properties of physiological electrical signals. Based on a deep learning signal processing model, feature representation training is performed on multimodal structural expression sequences to extract feature representation sequences that reflect the temporal structure and channel correlation characteristics of multimodal physiological electrical signals; During the feature representation training process, a structural reconstruction constraint is introduced to model the consistency between the feature representation sequence output by the model and the corresponding multimodal structural representation sequence, forming a feature representation sequence for signal generation.

[0019] Furthermore, addressing the issues of complex channel structure representation, unclear cross-modal correlation paths, and difficulty in maintaining temporal structural coherence in the deep feature learning stage of multimodal physiological electrical signals, this invention, after obtaining the fused multimodal structural expression sequence, first constructs a feature mapping relationship based on a deep learning signal processing model, mapping the multimodal structural expression sequence to a feature representation space to establish a joint expression framework for cross-modal temporal structures, thereby enhancing the model's ability to capture multimodal structural differences; subsequently, during the feature mapping process, feature representation training is performed on the multimodal structural expression sequence, extracting information reflecting the temporal structure of multimodal physiological electrical signals through internal model structure mapping. The feature representation sequence of channel correlation characteristics can form a continuous and stable cross-modal temporal feature trajectory. Based on this, a structural reconstruction constraint mechanism is introduced to perform reconstruction consistency modeling processing on the feature representation sequence and the multimodal structural expression sequence, maintaining the stability of the expression relationship between the model output structure and the original input structure, and improving the structural reliability of the deep representation. Finally, the feature representation sequence after reconstruction consistency modeling processing is used as the feature representation sequence for signal generation, forming a stable and reliable model input foundation for the subsequent decoding process, significantly improving the expression consistency and temporal continuity capability of the deep learning signal processing model in the process of multimodal physiological electrical signal structural modeling.

[0020] Preferably, multimodal physiological electrical signals exhibit significant structural fluctuations, prominent modal response differences, and complex cross-channel structural coupling relationships during the deep feature learning stage. This invention introduces a structural reconstruction constraint mechanism into the deep learning signal processing model. By maintaining structural consistency between the feature representation sequence and the multimodal structural expression sequence, the deep learning process can stably capture cross-modal feature expression rules and effectively suppress structural shifts during training. Under the structural reconstruction constraint mechanism, the model can uniformly express the channel response differences and intermodal temporal differences of multimodal physiological electrical signals. While maintaining model training stability, it improves the integrity of the feature representation sequence in terms of structural expression, providing reliable support for the final multimodal physiological electrical signal processing results and effectively enhancing the continuity of expression and output stability during the signal generation stage.

[0021] Preferably, in step S6, a deep learning-based multimodal physiological electrical signal processing model is constructed. The feature representation sequence formed by the multimodal structural expression sequence is used as input, the feature representation sequence is decoded and reconstructed, and the decoding and reconstruction result is output as the multimodal physiological electrical signal processing result.

[0022] Furthermore, addressing the characteristics of multimodal physiological electrical signals exhibiting significant temporal differences in modal response, complex channel structure correlations, and difficulty in stably expressing the continuity of state changes under joint acquisition backgrounds, this invention constructs a deep learning-based multimodal physiological electrical signal processing model. Using a multimodal physiological electrical signal dataset as input, it sequentially completes cross-modal time-delay collaborative embedding construction, perturbation coherent modulation, multi-scale differential modeling, and feature decoding reconstruction. By performing unified decoding mapping on the feature representation sequences formed from the multimodal structural expression sequences, a processing result output mechanism capable of stably reflecting the temporal structure, channel correlations, and state change characteristics of multimodal physiological electrical signals is constructed. This achieves unified modeling and continuous output of the structural characteristics and dynamic evolution process of multimodal physiological electrical signals, effectively improving the overall performance of multimodal physiological electrical signal processing results in terms of structural consistency, temporal coherence, and state expression stability.

[0023] Compared with existing technologies, this invention focuses on the temporal response differences and channel structure correlation characteristics of multimodal physiological electrical signals during joint acquisition and processing. It constructs a signal processing mechanism centered on cross-modal time delay modeling, perturbation coherent modulation, and multi-scale differential expression. By constructing a multimodal physiological electrical signal dataset and introducing intermodal intrinsic time delay relationships, it achieves a unified expression of the coordinated changes in different physiological modes over time, effectively improving the structural consistency of multimodal physiological electrical signals during complex state changes. In the cross-modal time delay distributed response construction stage, by forming cross-modal time delay coherent embedding sequences, it enhances the model's ability to characterize the sequential relationships and temporal correlation characteristics of different physiological modes' responses. In the dynamic coherent modulation stage, a perturbation coherence factor is introduced and combined with a time recursion mechanism to achieve coordinated modulation of the perturbation response of multi-channel physiological electrical signals, thereby improving the model's stability in expressing dynamic correlation changes between channels. In the multi-scale differential modeling stage, by constructing a multi-scale differential structure and generating state transition coding sequences, the model's ability to sensitively express the amplitude and transition characteristics of physiological electrical signal state changes is enhanced. Finally, by constructing a deep learning signal processing model and performing feature expression, structural reconstruction, and decoding reconstruction, a unified modeling and stable output of the structural characteristics and state change processes of multimodal physiological electrical signals are achieved, effectively improving the performance of multimodal physiological electrical signal processing results in terms of continuity, robustness, and consistency. Attached Figure Description

[0024] Figure 1 This is a flowchart of a multimodal physiological electrical signal processing method based on deep learning provided by the present invention.

[0025] Figure 2 This is a structural diagram of the multimodal physiological electrical signal dataset provided by the present invention.

[0026] Figure 3This is a structural diagram of constructing a cross-modal time delay distributed response structure and forming a cross-modal time delay cooperative embedding sequence in this invention.

[0027] Figure 4 This is a structural diagram of the present invention for calculating the perturbation of multi-channel physiological electrical signals and forming a dynamic coherent modulation sequence.

[0028] Figure 5 This is a structural diagram of constructing a multi-scale difference structure and forming a multimodal structural expression sequence provided by the present invention.

[0029] Figure 6 This is a structural diagram of the deep learning signal processing model and the training of feature representation and structural reconstruction provided by the present invention.

[0030] Figure 7 This is a comparison chart of the multimodal physiological electrical signal processing results provided by the present invention.

[0031] Figure 8 This is a visualization of the multimodal physiological electrical signal processing results provided by the present invention. Detailed Implementation

[0032] This invention provides a deep learning-based method for processing multimodal physiological electrical signals. Addressing the issues of differences in intrinsic time delay relationships between modes, time-varying channel perturbations, and inconsistent representation of state transitions in multimodal physiological electrical signals, this invention proposes a multi-stage processing mechanism that integrates cross-modal time delay distributed response structures, dynamic coherent modulation sequences, and state transition coding sequences. This mechanism includes constructing a cross-modal time delay distributed response structure to form a cross-modal time delay co-embedding sequence; introducing a perturbation coherence factor and generating a dynamic coherent modulation sequence through time recursion; and constructing a multi-scale differential structure based on the modulated cross-modal time delay co-embedding sequence and generating a state transition coding sequence. This achieves unified modeling and stable reconstruction output of the cross-modal time delay structure, channel perturbation coherent structure, and state transition structure of multimodal physiological electrical signals.

[0033] Please see Figure 1 As shown in the figure, a multimodal physiological electrical signal processing method based on deep learning is described in this application embodiment. The specific steps are as follows.

[0034] S1. Obtain the raw sequence data of multimodal physiological electrical signals, preprocess the raw sequence data, and construct a multimodal physiological electrical signal dataset.

[0035] The construction process of the multimodal physiological electrical signal dataset for multimodal physiological electrical signal processing includes three stages: physiological electrical signal acquisition, preprocessing, and structured organization. In the physiological electrical signal acquisition stage, a human multimodal physiological electrical signal acquisition system is used to continuously monitor the subjects and collect raw sequence data of physiological electrical signals under multiple physiological modalities. Each physiological modality corresponds to a physiological electrical signal channel, and the acquired physiological electrical signals change continuously over time to form a time series. All physiological electrical signal channels are acquired synchronously under a unified time reference, and modality identification information, signal channel number information, and time index information are recorded synchronously during the acquisition process. This method is used to clarify the correspondence between different physiological modalities, physiological electrical signal channels, and time series. In this embodiment, the sampling rate of the original physiological electrical signal is set to 250Hz, and the sampling precision is set to 16bit to ensure that the temporal and amplitude resolutions of the physiological electrical signal meet the requirements of subsequent processing. In the preprocessing stage, the original physiological electrical signal sequences acquired under different physiological modalities are first subjected to uniform resampling processing to maintain a consistent sampling rate and eliminate time scale deviations between different acquisition channels. Subsequently, bandpass filtering processing is performed on each physiological electrical signal channel, with the filter passband range set to 0.5Hz~45Hz. To suppress baseline drift and high-frequency noise interference; after filtering, baseline correction is performed on each physiological electrical signal channel to bring the mean of the signal in the time dimension to zero; further, the amplitude of the physiological electrical signal is normalized, and the normalized signal amplitude is uniformly mapped to the range of [−1,1] to eliminate the difference in amplitude dimensions between different physiological modes; subsequently, the physiological electrical signal sequence is segmented into fixed time windows along the time dimension, with the time window length set to 2s and the time window sliding step size set to 1s, thereby dividing the continuous physiological electrical signal sequence into several signal segments with temporal continuity; in the structured group In the weaving stage, the preprocessed physiological electrical signal segments are uniformly organized according to modal identification information, signal channel number information, and time index information. The multimodal physiological electrical signal segments under the same time index are aligned and arranged to construct a multimodal physiological electrical signal dataset containing a multimodal and multi-channel time series structure. The multimodal physiological electrical signal dataset is stored in tensor form, where the time dimension corresponds to the sequence order of the signal segments, the channel dimension corresponds to the physiological electrical signal channel number under different physiological modes, and the numerical elements correspond to the normalized physiological electrical signal amplitude, which serves as the input basis for the subsequent construction of the cross-modal time delay distribution response structure.

[0036] S2. Based on the multimodal physiological electrical signal dataset, construct a cross-modal time delay distribution response structure, introduce the intrinsic time delay relationship between modes, and form a cross-modal time delay collaborative embedding sequence.

[0037] Furthermore, in step S2, a cross-modal time delay distributed response structure is constructed and a cross-modal time delay cooperative embedding sequence is formed. The process is shown in Figure 2. The specific steps for constructing the cross-modal time delay distributed response structure and forming the cross-modal time delay cooperative embedding sequence are as follows.

[0038] S21. Based on the multimodal physiological electrical signal dataset, for each physiological mode's physiological electrical signal sequence, extract multiple sets of modality-related intrinsic time delays within a preset time range. The physiological electrical signal sequence of the m-th physiological mode is denoted as m(t), and its mathematical model is: ; in, Let be the physiological electrical signal amplitude of the c-th signal channel in the m-th physiological modality at time index t. M represents the number of signal channels corresponding to the m-th physiological modality, t is the time index and t∈{1,2,...,T}, T is the time series length, and M is the number of physiological modalities; in this embodiment, m=3 and T=200. Within a preset time range, multiple sets of modality-related intrinsic time delays are introduced for different physiological modes to describe the time propagation relationship between electrocardiographic excitation, blood pressure response, and peripheral blood flow reflex. The mathematical model is as follows: ; in, Let k be the sampling period, and k ∈ {1,2,...,K}. m}, Let be the number of intrinsic time delays corresponding to the m-th physiological mode; in this embodiment, =0.004s, = 48, corresponding to a maximum latency of 0.192s; S22. Based on the time offset relationship corresponding to different intrinsic time delays, perform time delay alignment processing on the physiological electrical signal sequences of each physiological mode to construct the cross-modal time delay distribution response structure corresponding to the physiological mode. Its mathematical model is: ; in, Let be the multi-channel physiological electrical signal vector of the m-th physiological mode at time index t. The intrinsic time delay of the m-th physiological mode The time delay aligns the physiological electrical signal quantity. It is a Euclidean norm. The weighting coefficients for the m-th physiological mode under the k-th intrinsic time delay are given; in this embodiment, =1 / 48; S23. Based on the cross-modal time delay distribution response structure corresponding to each physiological mode. The response results from different physiological modalities are collaboratively combined to form a cross-modal time-delay collaborative embedding sequence. Its mathematical model is: ; in, Let M be the modality weight corresponding to the m-th physiological modality, and M be the number of physiological modalities. In this embodiment, = 1 / 3.

[0039] S3. Based on the cross-modal time-delay collaborative embedding sequence, calculate the perturbation amount of the multi-channel physiological electrical signal, introduce the perturbation coherence factor, and form a dynamic coherent modulation sequence through time recursion to perform channel-level modulation processing on the cross-modal time-delay collaborative embedding sequence.

[0040] Furthermore, in step S3, the perturbation amount of the multi-channel physiological electrical signals is calculated and a dynamic coherent modulation sequence is formed, as follows: Figure 3 As shown, the specific steps for calculating the perturbation of multi-channel physiological electrical signals and forming a dynamic coherent modulation sequence are as follows.

[0041] S31. Based on the channel component notation of the cross-modal time-delay co-embedded sequence Z(t), and by calculating the difference between adjacent sampling times along the time dimension, the perturbation sequence is obtained. Its mathematical model is: ; in, Represents cross-modal delay co-embedding sequences The value of the c-th channel component at time index t, where time index t satisfies ∈ {2,3,...,T}; in this embodiment, T=200, t=2; S32. Based on the disturbance sequence For any channel pair Based on the difference in perturbation amounts between the two channels at the same time index t, a perturbation coherence factor is constructed. Its mathematical model is: ; Among them, c i With c j This represents any two different physiological signal channels under the same time index. This represents the difference in the amplitude of disturbances in different channels at the same time; in this embodiment, The range of values ​​is And when hour, =1; S33, Based on perturbation coherence factor The perturbation responses of all channels are normalized and weighted to form a channel-level perturbation coherent response sequence. Its mathematical model is: ; Among them, the summation symbol Represents cross-modal delay co-embedding sequences All channel indices c are aggregated for calculation. This indicates the weighting strength term used in this embodiment, which retains its sign and only squares the original weights to enhance the contribution of coherence differences. For the cth j The change in the amplitude of perturbation of each physiological signal channel at time index t; Coherent response sequence to channel-level perturbation The execution time recursive processing forms a dynamic coherent modulation sequence. Its mathematical model is: ; Where α is the time recursion coefficient; in this embodiment, α = 0.85, and the initial recursion value is set to... ; S34, Based on the dynamic coherent modulation sequence For the channel components of cross-modal time-delay co-embedding sequences Perform channel-level modulation processing to obtain a modulated cross-modal time-delay cooperative embedding sequence. Its mathematical model is: ; in, For modulated cross-modal time-delay cooperative embedding sequences, Represents cross-modal delay co-embedding sequences The component on channel c.

[0042] S4. Based on the modulated cross-modal time-delay co-embedding sequence, construct a multi-scale differential structure for the multi-channel physiological electrical signal, and generate a state transition coding sequence to characterize the channel change characteristics, thereby obtaining a multi-modal structure expression sequence for signal processing.

[0043] Furthermore, in step S4, a multi-scale difference structure is constructed and a multimodal structural expression sequence is formed, as follows: Figure 4 As shown, the specific steps for constructing a multi-scale difference structure and forming a multimodal structure expression sequence are as follows.

[0044] S41. Based on the modulated cross-modal time-delay cooperative embedding sequence Along the time dimension at different time scales s, the variation amplitude between adjacent sampling points of each physiological electrical signal channel is calculated, and a multi-scale difference structure is constructed. Its mathematical model is: ; Where s is the time scale length, t is the time index satisfying t>2s, and k is the discrete offset index at time scale s, k∈ {0,1,2,…,s−1}; in this embodiment... The corresponding sampling intervals are 0.004s, 0.012s, and 0.020s; S42, Based on multi-scale difference structure The differential responses at different time scales are weighted and fused to obtain the scale-fused jump intensity sequence. Its mathematical model is: ; in, This represents the difference magnitude of the fusion weights at scale s for different time scales; in this embodiment, =1 / 3; S43, Based on the scale fusion jump intensity sequence The threshold determination rule will be used to determine the threshold. Mapped to state transition coding sequence To achieve discrete encoding of state changes of different amplitudes in physiological electrical signals, the mathematical model is as follows: ; in, The threshold for determining the transition; in this embodiment, =0.15; S44. Encode the state transition sequences corresponding to each physiological electrical signal channel. With modulated cross-modal time-delay co-embedding sequences Joint organization is carried out to form multimodal structural expression sequences. Its mathematical model is: ; in, It is a multimodal structural expression sequence.

[0045] S5. Based on the multimodal structural expression sequence, construct a deep learning signal processing model, perform feature expression and structural reconstruction training, and form a feature representation sequence for signal generation.

[0046] Furthermore, in step S5, a deep learning signal processing model is constructed and feature representation and structural reconstruction training is performed, as follows: Figure 5 As shown, the specific steps for constructing a deep learning signal processing model and performing feature representation and structure reconstruction training are as follows.

[0047] S51. Express the sequence based on the fused multimodal structure. To construct a deep learning signal processing model, a multimodal structural representation sequence is used as the model input, and the model parameter set is used... Establish feature mapping relationships and generate intermediate feature mapping results. Its mathematical model is: ; in, This is the parameter set of a deep learning signal processing model, which performs a linear mapping on the multimodal structural representation sequence in matrix form. This is an element-wise algebraic coupling operation; S52. Based on the established feature mapping relationship, the intermediate feature mapping results are processed. Perform feature representation training to obtain the feature representation sequence output by the model. Its mathematical model is: ; in, Characteristic representation sequence The d-th feature component at time index t, where d∈{1,2,…,D}, t is the time index, t∈{1,2,…,T}, and T represents the sequence of multimodal structural expressions. The corresponding time index length; in this embodiment, D=128; S53. In the feature representation training process, a structural reconstruction constraint is introduced, which is achieved by modifying the feature representation sequence output by the model. With the corresponding multimodal structural expression sequence By performing algebraic superposition, a sequence of feature representations for signal generation that satisfies structural consistency is formed. Its mathematical model is: ; in, Q(t) constitutes the feature representation sequence used for signal generation, representing the updated intermediate representation results formed after introducing structural reconstruction constraints.

[0048] S6. Based on the deep learning signal processing model, perform decoding and reconstruction processing on the feature representation sequence formed in steps S2 to S5, and output the reconstruction result as the result of multimodal physiological electrical signal processing.

[0049] In step S6, for the deep learning signal processing model, the first input is the feature representation sequence obtained from the multimodal physiological electrical signal dataset processed through steps S2 to S5. Steps S2 to S5 are then executed sequentially, and decoding and reconstruction are completed. Specifically, in the cross-modal time-delay distributed response structure construction stage, a cross-modal time-delay distributed response structure is constructed based on the multimodal physiological electrical signal dataset. Intrinsic time-delay relationships between modes are introduced to form a cross-modal time-delay co-embedding sequence, providing time-delay consistency constraints for subsequent cross-modal co-expression. In the perturbation coherent modulation processing stage, the perturbation amount of the multi-channel physiological electrical signals is calculated based on the cross-modal time-delay co-embedding sequence. A perturbation coherence factor is introduced, and a dynamic coherent modulation sequence is formed through time recursion. This sequence is then used to process the cross-modal time-delay co-embedding sequence. The process involves channel-level modulation to obtain a perturbation-coherently modulated cross-modal feature sequence. In the multi-scale differential structure and state transition coding stage, a multi-scale differential structure is constructed for the multi-channel physiological electrical signal based on the modulated cross-modal feature sequence, and a state transition coding sequence is generated to characterize the channel change characteristics, thus obtaining a multi-modal structure representation sequence for signal processing. Subsequently, in the feature representation and structure reconstruction training stage, a deep learning signal processing model is constructed based on the multi-modal structure representation sequence, and feature representation and structure reconstruction training is performed to form a feature representation sequence for signal generation. By integrating the above processes end-to-end, the feature representation sequences formed in steps S2 to S5 are finally decoded and reconstructed, and the reconstruction result obtained from the decoding and reconstruction is output as the multi-modal physiological electrical signal processing result.

[0050] Further, in step S6, a deep learning signal processing model is constructed, and based on this model, the feature representation sequences formed in steps S2 to S5 are decoded and reconstructed. The deep learning signal processing model is written in Python and implemented using the PyTorch deep learning framework. During model training, the Adam optimizer is used to update the parameters of the deep learning signal processing model, with an initial learning rate of 0.0003, a batch size of 64, and a total number of training rounds of 200. Through multiple rounds of iterative training of the deep learning signal processing model, the feature representation sequences formed based on the multimodal structural expression sequences are gradually decoded and reconstructed. The overall training state of the model tends to stabilize around the 160th round. Through the above decoding and reconstruction process, the reconstruction result is output as the multimodal physiological electroencephalogram (EEG) signal processing result.

[0051] Furthermore, in step S6, the multimodal physiological electrical signal dataset is input into a deep learning signal processing model for processing. A comparison of the multimodal physiological electrical signal processing results is shown in the figure below. Figure 7As shown in the figure, the original multimodal physiological electrical signals exhibit amplitude fluctuations and disturbances under multimodal and multichannel conditions. However, the multimodal physiological electrical signals processed after decoding and reconstruction maintain the main waveform trends and suppress channel-level disturbance components under the same time axis alignment. The visualization of the multimodal physiological electrical signal processing results is shown in the figure below. Figure 8 As shown in the figure, the horizontal axis represents time in seconds, and the vertical axis represents the signal amplitude. It can be seen from the figure that the multimodal physiological electrical signal processing results present a continuous and stable waveform structure in multimodal multichannels, and reflect the structural consistency under the constraints of cross-modal time delay collaborative embedding sequence, dynamic coherent modulation sequence and multimodal structural expression sequence. The experimental results verify the stability and consistency of the deep learning signal processing model for the output of multimodal physiological electrical signal processing results.

[0052] The above description is only a preferred embodiment of the present invention. Those skilled in the art can make several substitutions, adjustments and combinations without departing from the concept of the present invention, and these modifications and improvements should be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based multi-modal physiological electrical signal processing method, characterized in that, Includes the following steps: S1. Obtain the raw sequence data of multimodal physiological electrical signals, preprocess the raw sequence data, and construct a multimodal physiological electrical signal dataset; S2. Based on the multimodal physiological electrical signal dataset, construct a cross-modal time-delay distributed response structure, introduce the intrinsic time-delay relationship between modes, and form a cross-modal time-delay collaborative embedding sequence; S3. Based on the cross-modal time-delay collaborative embedding sequence, calculate the perturbation amount of the multi-channel physiological electrical signal, introduce the perturbation coherence factor, and form a dynamic coherent modulation sequence through time recursion to perform channel-level modulation processing on the cross-modal time-delay collaborative embedding sequence. S4. Based on the modulated cross-modal time-delay co-embedding sequence, construct a multi-scale differential structure for multi-channel physiological electrical signals and generate a state transition coding sequence to characterize the channel change characteristics, thereby obtaining a multi-modal structure expression sequence for signal processing. S5. Based on the multimodal structural expression sequence, construct a deep learning signal processing model, perform feature expression and structural reconstruction training, and form a feature representation sequence for signal generation; S6. Based on the deep learning signal processing model, perform decoding and reconstruction processing on the feature representation sequence formed in steps S2 to S5, and output the reconstruction result as the result of multimodal physiological electrical signal processing.

2. The deep learning-based multi-modal physiologic electrical signal processing method of claim 1, wherein, The process of constructing a multimodal physiological electrical signal dataset includes: acquiring raw physiological electrical signal data output by a human multimodal physiological electrical signal acquisition system, which includes physiological electrical signal sequences of multiple signal channels changing over time under different physiological modes; The modal identification information, signal channel number information and corresponding time index information are acquired synchronously with the physiological electrical signals, which are used to characterize the correspondence between different physiological modalities, different signal channels and time series; The original physiological electrical signal sequences, modality identification information, signal channel number information, and time index information are organized to construct a multimodal physiological electrical signal dataset containing multimodal and multi-channel time series structures.

3. The deep learning-based multi-modal physiologic electrical signal processing method of claim 2, wherein, The construction process of the cross-modal time delay distribution response structure includes: based on the multimodal physiological electrical signal dataset, for each physiological mode, extracting multiple sets of modality-related intrinsic time delays within a preset time range for the physiological electrical signal sequence, and forming the time delay distribution response corresponding to the physiological mode based on the time offset relationship corresponding to different intrinsic time delays; Based on the time delay distribution response of each physiological mode, and based on the intrinsic time delay of each physiological mode, the time delay correlation between physiological modes is introduced, and the time delay distribution response under different physiological modes is synergistically combined to obtain the cross-modal time delay distribution response structure. Based on the cross-modal time-delay distribution response structure, multimodal physiological electrical signal sequences are weighted and combined to form cross-modal time-delay collaborative embedding sequences.

4. The deep learning-based multi-modal physiologic electrical signal processing method of claim 3, wherein, Based on the cross-modal time-delay co-embedding sequence, the signal difference of each physiological electrical signal channel at adjacent time points is calculated along the time dimension to obtain the perturbation sequence of the corresponding channel; Based on the perturbation sequence of each channel, the perturbation difference between any pair of channels is calculated, and a perturbation coherence factor is introduced according to the perturbation difference to weight and combine the perturbation responses of multiple channels to form a channel-level perturbation coherence response sequence. Time recursion processing is performed on the channel-level perturbation coherent response sequence to construct a dynamic coherent modulation sequence that evolves over time. The dynamic coherent modulation sequence is then used to perform channel-level modulation on the cross-modal time-delay co-embedding sequence to obtain the perturbation-coherently modulated cross-modal feature sequence.

5. The deep learning-based multi-modal physiologic electrical signal processing method of claim 4, wherein, Based on the cross-modal feature sequence after perturbation coherent modulation, a multi-scale differential structure is constructed for each physiological electrical signal channel along the time dimension. The signal change amplitude between adjacent sampling points is calculated at different time scales to form a multi-scale differential response sequence. Based on the multi-scale differential response sequence, the differential responses at each time scale are weighted and fused to obtain a scale fusion jump intensity sequence that characterizes the intensity of changes in the state of physiological electrical signals. Based on the scale fusion jump intensity sequence, the state changes of the physiological electrical signal channel are discriminated at each time index to form the corresponding state jump coding sequence. The state jump coding sequence is determined by the numerical relationship between the scale fusion jump intensity and the preset judgment threshold, and different discrete coding states are taken under different numerical relationships. These are organized as discrete state components in the multimodal structure expression sequence to obtain the multimodal structure expression sequence for subsequent signal processing.

6. The deep learning-based multi-modal physiologic electrical signal processing method of claim 5, wherein, Based on the fused multimodal structural expression sequence, a deep learning signal processing model is constructed. The multimodal structural expression sequence is used as the model input to establish a feature mapping relationship for characterizing the structural properties of physiological electrical signals. Based on a deep learning signal processing model, feature representation training is performed on multimodal structural expression sequences to extract feature representation sequences that reflect the temporal structure and channel correlation characteristics of multimodal physiological electrical signals; During the feature representation training process, a structural reconstruction constraint is introduced to model the consistency between the feature representation sequence output by the model and the corresponding multimodal structural representation sequence, forming a feature representation sequence for signal generation.

Images