Multimodal physiological signal depression state recognition system based on data enhancement

Abstract

The invention discloses a multi-modal physiological signal depression state recognition system based on data enhancement, and the system comprises a recognition module which is configured to input a multi-modal physiological signal sample into a trained depression state recognition model, and obtains a depression state recognition result; wherein the trained depression state recognition model is used for performing feature extraction on each preprocessed modal physiological signal to obtain a feature corresponding to each modal physiological signal; performing feature fusion on the features corresponding to the multi-modal physiological signals to obtain fused features; the fusion features are classified, and a depression state recognition result is obtained; wherein in the training process of the depression state recognition model, a training set is preprocessed, then data enhancement is performed on a preprocessing result, then the model is trained by adopting enhanced data, and when the total loss function value of the model does not decrease any more, training is stopped, and the trained depression state recognition model is obtained.

Description

Technical Field

[0001] This invention relates to the field of depression diagnosis and identification technology, and in particular to a depression state identification system based on data-enhanced multimodal physiological signals. Background Technology

[0002] Depressive disorders are a common group of mental illnesses, typically characterized by persistent low mood, loss of interest, or anhedonia. Severe cases may be accompanied by risky behaviors such as self-harm and suicide, placing a significant burden on individuals, families, and society. Timely assessment and intervention are crucial prerequisites for the treatment of depressive disorders. However, in clinical practice, the diagnosis and severity assessment of depressive disorders are usually based on the DSM-related diagnostic criteria combined with the ICD coding system, while also considering interviews, scales, and physician experience. This approach is easily influenced by factors such as the subject's subjective statements, level of cooperation, and differences in the assessor's experience.

[0003] With the development of wearable sensors and intelligent computing, the use of objective physiological signals for auxiliary assessment of depression has gradually attracted attention. Compared with overt behavioral signals such as facial expressions and voice, peripheral physiological signals such as electrocardiogram (ECG), pulse / photoplethysmography (PPG), and respiratory signal (RSP) have the advantages of continuous acquisition, difficulty in subjective spoofing, and relatively low acquisition cost. They can provide objective information related to autonomic nervous system regulation from the perspective of cardiovascular activity and respiratory regulation. Furthermore, some studies have attempted to achieve automatic identification of depressive disorders based on cardiovascular and respiratory physiological signals, indicating that this technical approach has certain feasibility. However, existing depressive state identification systems based on physiological time-series signals such as ECG, PPG, and RSP still have the following shortcomings: (1) In daily or wearable data collection scenarios, ECG, PPG and RSP are easily affected by motion artifacts, poor sensor contact and environmental noise, often resulting in problems such as noise superposition, baseline drift, amplitude abrupt change and local missing, which reduces the stability of feature extraction and affects the robustness of the recognition model.

[0004] (2) In actual acquisition, multimodal signals often have problems such as sampling rate differences, inconsistent start / stop times and time alignment errors. If there is a lack of a unified synchronization alignment and segmentation strategy, it is difficult to ensure that the three signals in the same training sample correspond one-to-one in the same time window, thereby weakening the effect of multimodal fusion modeling.

[0005] (3) The four-class classification task of depression usually has the problems of limited sample size and class imbalance, especially the moderate and severe samples are relatively scarce. Directly training deep models is prone to overfitting and weakening the ability to identify minority classes. To alleviate the problems of insufficient data and class imbalance, generative data augmentation can be used to expand the training set. However, in the multi-class conditional generation scenario, it still faces challenges such as unstable training and difficulty in ensuring the physiological rationality of generated samples and the consistency of labels. If there is a lack of stable training mechanism and sample selection strategy, low-quality samples may be introduced and affect the generalization performance of the model. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a data-augmented multimodal physiological signal depression state recognition system. This system, designed for ECG, PPG, and RSP trimodal peripheral physiological signals, improves the robustness and generalization ability of the model under noise interference and class imbalance conditions through standardized preprocessing, unified sampling rate alignment, synchronous segmentation, and stable and effective generative data augmentation and multimodal fusion modeling strategies. This enables reliable differentiation of normal, mild, moderate, and severe depression states.

[0007] On the one hand, a data-enhanced multimodal physiological signal-based system for identifying depressive states is provided, including: The acquisition module is configured to acquire multimodal physiological signal data to be identified. The preprocessing module is configured to preprocess the physiological signal data of each modality to obtain preprocessed physiological signal samples of each modality. The identification module is configured to: input multimodal physiological signal samples into a trained depression state identification model to obtain a depression state identification result; wherein, the trained depression state identification model is used to extract features from each preprocessed modality of physiological signal to obtain features corresponding to each modality of physiological signal; fuse the features corresponding to the multimodal physiological signals to obtain fused features; and classify the fused features to obtain a depression state identification result. In the training process of the depression state recognition model, a training set is used for training. The training set includes multimodal physiological signal data with known depression state labels. During the training process, the training set is first preprocessed, and then the preprocessed results are augmented with data. Next, the augmented data is used to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and the trained depression state recognition model is obtained.

[0008] The above technical solution has the following advantages or beneficial effects: (1) By inputting the three-modal peripheral physiological signals of ECG, PPG and RSP into the recognition model based on deep neural network, the results of the four-level depression state can be automatically output, thus providing a more objective auxiliary basis for depression assessment. Furthermore, studies have shown that the use of cardiovascular and respiratory physiological signals for automatic depression recognition is feasible.

[0009] (2) By preprocessing and synchronously segmenting the three-mode signals and performing feature-level fusion, it is beneficial to reduce the impact of noise interference and alignment error on the modeling results and improve the stability and robustness of the recognition model in daily acquisition scenarios.

[0010] (3) By introducing generative data augmentation, it is beneficial to alleviate the problems of limited sample size and class imbalance. At the same time, the use of gradient-penalized Wasserstein adversarial training and PBT online parameter tuning strategy is beneficial to improve the training stability of generative models and the availability of generated samples, thereby improving the generalization performance of downstream depression state recognition models.

[0011] (4) The feature extraction module of this invention uses an MPARN+Bi-LSTM structure to learn features for ECG, PPG and RSP respectively. MPARN extracts temporal features of different scales through multi-scale convolutional paths and introduces a path attention mechanism to adaptively weight features of different convolutional paths, thereby highlighting scale features with greater information content and suppressing redundant scale features. At the same time, MPARN introduces residual connections to improve the network training stability and feature expression ability. On this basis, Bi-LSTM uses the bidirectional contextual information of the sequence for temporal modeling, which is beneficial to obtain more complete temporal dependencies and improve the ability to distinguish physiological temporal signals.

[0012] (5) The present invention uses a Transformer encoder to model the fusion features, which can learn the global dependency relationship of the fusion features through the self-attention mechanism and support parallel computing to improve training and inference efficiency, thereby helping to further improve the recognition effect of the four-class classification of depression.

[0013] (6) By using a composite loss function to jointly consider factors such as classification accuracy, class imbalance and prediction constraints, it is beneficial to improve the accuracy of depression state recognition and training stability. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0015] Figure 1 This is a schematic diagram of the depressive state recognition system according to an embodiment of the present invention; Figure 2 This is a flowchart of the data enhancement module according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the feature extraction process according to an embodiment of the present invention; Figure 4 This is a flowchart of the electrocardiogram signal filtering process according to an embodiment of the present invention; Figure 5 This is a flowchart of pulse signal filtering according to an embodiment of the present invention; Figure 6 This is a flowchart of the respiratory signal filtering process according to an embodiment of the present invention; Figures 7(a) and 7(b) are the original electrocardiogram (ECG) signal and the filtered signal, respectively, according to an embodiment of the present invention. Figures 8(a) and 8(b) are the original pulse signal and the filtered signal of an embodiment of the present invention. Figures 9(a) and 9(b) are the original respiratory signal and the filtered signal of an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0017] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0018] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0019] To address the shortcomings in robustness and generalization of existing technologies for identifying depressive states based on peripheral physiological time-series signals, which are susceptible to noise interference, time alignment difficulties, limited sample size, and class imbalance, this invention provides a depressive state identification system based on electrocardiogram (ECG), pulse / photoplethysmography (PPG), and respiratory signal (RSP). The identification results include four states: normal, mild depression, moderate depression, and severe depression.

[0020] Example 1 like Figure 1 As shown, this embodiment provides a data-enhanced multimodal physiological signal-based system for identifying depressive states, including: The acquisition module is configured to acquire multimodal physiological signal data to be identified. The preprocessing module is configured to preprocess the physiological signal data of each modality to obtain preprocessed physiological signal samples of each modality. The identification module is configured to: input multimodal physiological signal samples into a trained depression state identification model to obtain a depression state identification result; wherein, the trained depression state identification model is used to extract features from each preprocessed modality of physiological signal to obtain features corresponding to each modality of physiological signal; fuse the features corresponding to the multimodal physiological signals to obtain fused features; and classify the fused features to obtain a depression state identification result. In the training process of the depression state recognition model, a training set is used for training. The training set includes multimodal physiological signal data with known depression state labels. During the training process, the training set is first preprocessed, and then the preprocessed results are augmented with data. Next, the augmented data is used to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and the trained depression state recognition model is obtained.

[0021] Furthermore, the multimodal physiological signal data includes: electrocardiogram (ECG), pulse rate (PPG), and respiratory rate (RSP). ECG, PPG, and RSP are synchronously acquired using a multi-channel physiological signal acquisition system to ensure the spatiotemporal synchronization of the three physiological signals.

[0022] Current medical and signal processing research indicates that depression causes changes in the function of the human autonomic nervous system, which are directly reflected in cardiovascular and respiratory activities. Depressed individuals typically exhibit decreased heart rate variability and weakened autonomic nervous system regulation. These physiological changes can be objectively reflected through electrocardiogram (ECG), pulse rate (PPG), and respiratory rate (RSP) signals.

[0023] For example, acquiring multimodal physiological signal data refers to the subject's electrocardiogram (ECG), pulse rate (PPG), and respiratory rate (RSP) signals in a resting state. The three modalities of the subject's resting state are simultaneously acquired using physiological signal acquisition equipment. During signal acquisition, it is preferable to ensure a relatively quiet acquisition environment with minimal interference, and to minimize the impact of noise, electromagnetic interference, and significant body movement on signal quality.

[0024] For example, the RM6280C multi-channel physiological data acquisition and processing system is used as the acquisition platform to achieve synchronous acquisition and recording of ECG, PPG and RSP.

[0025] For example, the ECG signal is obtained by arranging electrodes in a standard lead II or equivalent lead configuration to obtain a stable electrocardiogram waveform; the PPG signal is acquired at the fingertip (e.g., the left index finger) through a photoelectric pulse transducer to reflect the periodic changes in peripheral vascular volume with the cardiac cycle; the RSP signal is acquired through a respiratory sensor fixed to the chest and abdomen to record displacement / deformation changes caused by respiratory movements.

[0026] For example, the instrument sampling rate can be set to 1 kHz, the sensitivity can be adjusted to 10 mV / division, the scan speed can be set to 200 ms / division, and the acquisition duration can be set to 330 seconds. To ensure signal integrity and validity, the hardware filtering function of the acquisition system can be disabled to retain the original waveform information for noise reduction in the subsequent preprocessing module. If any sudden abnormal situation occurs during the experiment (such as the subject moving their body, the sensor becoming loose, or frequent deep breathing causing abnormal respiratory signals), the acquisition can be interrupted and data acquisition can be resumed with the subject's consent. After the acquisition is completed, the acquired experimental data is processed and saved using a computer.

[0027] Furthermore, the preprocessing of each modality of physiological signal data to obtain preprocessed samples of each modality of physiological signal includes: downsampling, filtering, and normalization.

[0028] Furthermore, the downsampling process is used to uniformly reduce the sampling rate of ECG, PPG and RSP signals from the original 1000Hz to a preset sampling rate of 256Hz, so as to reduce the amount of computation and ensure the consistency of the three-modal data in the time dimension.

[0029] Furthermore, the filtering process includes: filtering the electrocardiogram signal, filtering the pulse signal, and filtering the respiration signal.

[0030] Furthermore, the filtering process employs wavelet threshold denoising to filter the ECG signal, PPG signal, and RSP signal respectively, in order to reduce the impact of baseline drift and various types of noise on subsequent identification.

[0031] For example, the basic principle of wavelet threshold denoising is as follows: first, the original signal is decomposed into wavelet coefficients at each scale; then, the wavelet coefficients are shrunk / filtered according to the threshold function; and finally, the filtered signal is reconstructed based on the processed wavelet coefficients.

[0032] Wavelet thresholding is used to denoise signals. Wavelet transform can better characterize non-stationary signals, especially the detailed features of physiological signals. The basic principle of wavelet thresholding is to first decompose the original signal using wavelet transform to obtain wavelet coefficients at each level, then filter the wavelet coefficients according to the selected threshold, and finally reconstruct the filtered signal based on the filtered wavelet coefficients.

[0033] The basic principle of wavelet transform is to convolve a fundamental wavelet function, which can be scaled and time-shifted, with the original signal to obtain the frequency characteristics of the original signal at different time points. Let φ(t) be a fundamental wavelet function, and a series of sub-wavelets can be generated according to the following formula:

[0034] Where 'a' is called the scaling coefficient, a parameter representing frequency, and 'b' is called the translation coefficient, a parameter representing time. According to this family of wavelet functions, the wavelet transform formula for the original signal f(t) is:

[0035] in, These are called wavelet coefficients. Because this method is computationally too intensive, the scaling coefficients *a* of the basic wavelet function are generally discretized using a power series to the base 2. The process of analyzing the original signal using these discrete wavelets is called Discrete Wavelet Transform (DWT). In sub-wavelets, the scaling parameter increases by a multiple of 2, so the detectable frequencies corresponding to the sub-wavelet decrease by a multiple of 1 / 2. The spectrum corresponding to the basic wavelet is located at the high end of the entire signal spectrum, occupying half of the total spectrum. Other sub-wavelets, due to the increase in the scaling parameter, shift their spectra towards the low-frequency end of the spectrum, and the frequency range they cover decreases accordingly by a multiple of 1 / 2. Ideally, the spectrum ranges of all wavelets overlap and overlap.

[0036] In wavelet thresholding denoising, the selection of the threshold and the threshold function are the most crucial steps. The threshold used in this invention is the VisuShrink threshold, and its formula is as follows:

[0037] in Here, represents the standard deviation of the noise, and n represents the signal length. Generally, the median of the wavelet coefficients representing high frequencies is divided by 0.6745 as an estimate of the noise standard deviation. A soft thresholding function is used, which yields wavelet coefficients with better continuity; its formula is as follows:

[0038] These are wavelet coefficients. The threshold value is used.

[0039] Furthermore, such as Figure 4 As shown, the filtering process for the electrocardiogram signal includes: Discrete wavelet decomposition and reconstruction were performed using the sym8 wavelet basis, with a fixed decomposition level of 8. VisuShrink threshold and soft threshold functions were used to perform threshold denoising on the coefficients to obtain the filtered electrocardiogram signal.

[0040] The effective frequency components of electrocardiogram signals are usually distributed in the range of 0.5Hz to 150Hz, and in actual analysis, the range of 0.5Hz to 40Hz is often used for denoising and feature extraction to suppress baseline drift and high-frequency interference.

[0041] Baseline drift in electrocardiogram (ECG) signals is usually low-frequency noise, with typical frequency components around 0.5 Hz. Therefore, when performing wavelet threshold denoising on ECG signals, the low-frequency trend term can be considered as one of the noise components that need to be suppressed.

[0042] To reduce residual spikes after threshold denoising and to preserve morphological features such as QRS as much as possible, the reconstructed ECG signal can be further smoothed using a Savitzky-Golay (SG) filter.

[0043] It should be understood that the basic principle of the SG filter is as follows: within a sliding window of length (2M+1), a p-order polynomial is fitted to the discrete sampling points within the window using the least squares method, and the value of the fitted polynomial at the center point of the window is used as the filter output. Therefore, the SG filter can be equivalently represented as a convolutional form:

[0044] in, The convolution coefficients are determined by the least squares polynomial fitting, x[n] is the input sequence, and y[n] is the output sequence. (In this embodiment, the SG filter parameters are fixed, for example: the window length is fixed at 11 points (2M+1=11), and the polynomial order is fixed at 3rd order (p=3).) Furthermore, such as Figure 5 As shown, the filtering process for the pulse signal includes: Discrete wavelet decomposition and reconstruction were performed using the db4 wavelet basis. The number of decomposition layers was fixed at 5. The coefficients were then reconstructed after threshold denoising using the VisuShrink threshold and soft threshold functions to obtain the filtered pulse signal.

[0045] The spectral information of pulse signals is mostly concentrated in the lower frequency band. The main peak corresponding to heart rate is usually around 1 Hz, and the effective bandwidth for heart rate correlation analysis is usually less than 5 Hz.

[0046] For example, in order to match the effective frequency band of the pulse signal and suppress low-frequency drift and high-frequency noise, the pulse signal often uses a bandpass range of 0.5Hz to 4Hz or 0.5Hz to 5Hz as the signal focus band.

[0047] Furthermore, such as Figure 6 As shown, the filtering process for the respiratory signal includes: Discrete wavelet decomposition and reconstruction were performed using the db8 wavelet basis, with a fixed number of decomposition layers of 8. The coefficients were then reconstructed after threshold denoising using the VisuShrink threshold and soft threshold functions to obtain the filtered respiratory signal.

[0048] The main frequency components of resting breathing in healthy adults typically fall within the range of 0.1 Hz to 0.5 Hz, therefore the focus band of RSP can be set to 0.1 Hz to 0.5 Hz.

[0049] To further suppress high-frequency noise components in the RSP, the reconstructed respiratory signal RSP can be filtered using a Butterworth low-pass filter to obtain the final RSP denoising result.

[0050] It should be understood that the amplitude-frequency response of the Butterworth filter is as flat as possible within the passband, and the squared amplitude response of its nth-order analog low-pass filter can be expressed as:

[0051] in, Let the filter order be . The cutoff angular frequency (approximately corresponding to) (3dB cutoff frequency). (In this embodiment, the Butterworth low-pass filter parameters are fixed, for example: the order is fixed at 4th order (n=4), and the cutoff frequency is fixed at 0.5Hz, in order to cover the breathing concern band from 0.1Hz to 0.5Hz and suppress higher frequency noise.) Figures 7(a) and 7(b) are the original electrocardiogram signals and the filtered signals of the embodiment of the present invention; Figures 8(a) and 8(b) are the original pulse signals and the filtered signals of the embodiment of the present invention; Figures 9(a) and 9(b) are the original respiratory signals and the filtered signals of the embodiment of the present invention.

[0052] Furthermore, the normalization process employs min-max normalization to scale the amplitude of the filtered signal, thereby reducing amplitude differences caused by different individuals, different modes, and different acquisition conditions; for any mode signal sequence Normalized sequence for:

[0053] in, For sequence The minimum value within the current time window For sequence The maximum value within the current time window.

[0054] Furthermore, the trained depression state recognition model includes: a first feature extraction unit, a second feature extraction unit, and a third feature extraction unit; The first feature extraction unit is used to input the preprocessed electrocardiogram (ECG) signal, extract features from the ECG signal to obtain ECG features, and input the extracted ECG features into the fusion unit; The second feature extraction unit is used to input the preprocessed pulse signal, extract features from the pulse signal to obtain pulse features, and input the extracted pulse features into the fusion unit; The third feature extraction unit is used to input the preprocessed respiratory signal, extract features from the respiratory signal to obtain respiratory features, and input the extracted respiratory features into the fusion unit; The fusion unit fuses the three input features to obtain fused features; The classifier classifies the fused features to obtain the classification result.

[0055] Furthermore, such as Figure 3 As shown, the internal structures of the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit are identical. The first feature extraction unit includes a multi-path adaptive residual network (MPARN) and a Bi-LSTM network connected in sequence.

[0056] Furthermore, such as Figure 3 As shown, the feature extraction module is used to extract modal features from the preprocessed and segmented three-modal physiological signals, including: extracting ECG modal features from ECG signals, extracting PPG modal features from PPG signals, and extracting RSP modal features from RSP signals.

[0057] The feature extraction module includes an MPARN feature extraction network and a Bi-LSTM temporal modeling network; wherein, MPARN is used to extract multi-scale local features for each one-dimensional temporal signal and perform path attention weighted fusion, and Bi-LSTM is used to perform bidirectional temporal dependency modeling on the feature sequence output by MPARN.

[0058] Multi-Path Adaptive Residual Network (MPARN) is a deep model architecture that combines multi-scale feature extraction, path adaptive fusion, and residual learning. Its core idea is that input features are convolved on multiple parallel paths, each path focusing on information at different scales or frequencies. Then, an adaptive mechanism is used to weight and fuse the features from each path, and residual connections are used to improve network training stability and the completeness of feature representation. MPARN is a multi-scale attention recurrent network designed for time-series physiological signals. Its structure is customized and optimized for the rhythmicity, hierarchy, heterogeneity, and cross-temporal dependence of physiological signals.

[0059] It should be understood that the multi-path adaptive residual network includes a multi-path feature extraction module, an adaptive feature fusion module, and a residual connection module; The multi-path feature extraction module includes: convolutional layers C1, C2, and C3 arranged in parallel; the kernel size, dilation rate, or sampling scale of convolutional layers C1, C2, and C3 are different; single-modal physiological signals are input into convolutional layers C1, C2, and C3 respectively, and each convolutional layer C1, C2, and C3 outputs its own convolution result; the three convolution results are weighted and fused, and the weighted fused result is summed with the original input to obtain the feature extraction result of the multi-path adaptive residual network.

[0060] (1) Multi-path feature extraction module Furthermore, given the input sequence x∈R T×C MPARN first allocated it to Parallel feature pathways:

[0061] in, Indicates the first The convolution operations on each path have different kernel sizes, dilation rates, or sampling scales. Paths with small kernels focus on capturing local details, while paths with large kernels or dilated convolutions focus on capturing long-term dependencies. Medium-scale convolution paths, on the other hand, focus more on extracting signal rhythmicity and performing mid-frequency analysis.

[0062] No. The convolutional feature extraction process for each path can be represented as:

[0063] in, It is a non-linear activation function. Represents one-dimensional convolution. Let be the convolution kernel for path k.

[0064] (2) Adaptive feature fusion module The features generated by the multi-path structure are complementary, but the importance of different paths to the final task varies with the input. Therefore, MPARN introduces an adaptive fusion module that performs a weighted summation of the outputs of each path using learnable weights. First, the attention weights for each path are calculated:

[0065] Here, g() is a lightweight mapping (such as global pooling + fully connected layer). satisfy , indicating path The importance of the output is then used to obtain the fused features. .

[0066] (3) Residual connectivity and deep feature transfer mechanism MPARN introduces residual connections to superimpose the original input with the fused features, ensuring information flow and training stability. The formula is as follows:

[0067]

[0068] This residual design preserves the original temporal structure of the signal, preventing deep networks from damaging the signal's physiological characteristics; it also improves gradient flow during training, reducing the risk of gradient vanishing; and it allows features to retain their basic form in complex signals while simultaneously superimposing multi-scale abstract features. Residual output. It can be used as input to the next layer of the MPARN module, enabling the entire network to stack multiple levels of structure while maintaining stable training.

[0069] It should be understood that the Bi-LSTM is used for bidirectional time-series modeling of the feature sequences output by MPARN.

[0070] It should be understood that LSTM networks are a special type of recurrent neural network (RNN) that alleviates the long-term dependency problem that is common in RNNs to some extent. LSTM can effectively transmit and express information in long-term sequences without ignoring or forgetting useful information from long ago. It can better handle the dependencies of long sequences and avoid the gradient vanishing or exploding problem that occurs as the sequence length increases.

[0071] LSTM uses gate control structures and cell states to process information across different time steps. Each LSTM module includes a cell state and three gate structures: a forget gate, an input gate, and an output gate. The cell state stores the current LSTM state information and passes it to the next time step. The three gates are used to select which information can be remembered and passed down, and which information needs to be forgotten. Specifically: The forget gate determines which memories to forget based on the new input and the output from the previous time step, using the following formula:

[0072] Where h t-1 X represents the output at the previous time step. t Let W represent the input signal at the current moment, σ represent the sigmoid activation function layer, and W represent the input signal at the current moment. f and b f This represents the training parameters and bias parameters of the forget gate.

[0073] The input gate functions as the opposite of the forget gate; it determines the new input signal X. t and the output h of the previous time step t-1 Which information will be retained? The following formula can be used to describe the forget gate:

[0074] The output gate determines the output signal of the LSTM module at that moment, and its formula is as follows:

[0075] However, there is a problem with using LSTM models: they cannot encode information from back to front. Therefore, it is necessary to use bidirectional long short-term memory (Bi-LSTM) networks to capture bidirectional correlation information.

[0076] Among them, MPARN is used to extract multi-scale path features from physiological time-series signals and adaptively weight features of different scales through path attention, while using residual connections to improve training stability; Bi-LSTM is used to perform bidirectional temporal modeling of the feature representation to obtain more comprehensive temporal dependency features.

[0077] The classifier comprises a Transformer encoder and a classification output layer connected in sequence. The classification output layer comprises a global pooling layer and a fully connected layer connected in sequence. The depression state classification results include normal, mild depression, moderate depression, and severe depression.

[0078] Furthermore, the Transformer encoder is used to perform global dependency modeling on the fused features to obtain semantic feature representations; let the fused features be a sequence X=[x1,x2,…,x…]. T ], where x t Let be the fused feature vector at time step t; obtain the query matrix Q, key matrix K, and value matrix V through linear mapping, and calculate the attention output based on scaled dot product self-attention:

[0079] in, The dimension of the key vector.

[0080] The Transformer encoder employs a multi-head attention mechanism to model the correlation between different subspaces in parallel. Q, K, and V are projected as h heads respectively, and the attention output of each head is calculated and then concatenated. The multi-head attention output is obtained through a linear layer.

[0081] The Transformer encoder employs residual connections and layer normalization after both the attention sublayer and the feedforward network sublayer to improve training stability; the feedforward network is a position-wise feedforward network that performs a nonlinear transformation on the feature vector at each time step and outputs the encoded features.

[0082] To enable the model to perceive sequence position information, position encoding can be introduced into the input sequence. The position encoding is then added to the input features and used as the input to the Transformer encoder, as shown in the following formula:

[0083] in, This indicates the position of a word in a sentence, where 'd' represents the dimension of the positional encoding, 2. i The dimension representing an even number, 2 i +1 indicates an odd dimension (i.e., ... , ).

[0084] The classification output layer is used to map the semantic features output by the Transformer encoder to the predicted probabilities of four depression state categories. The output of the Transformer encoder is pooled to obtain a global feature vector r, and then logits are obtained through a fully connected layer: s = W·r + b. Finally, the four-class probability vector p = softmax(s) is obtained through the Softmax function, where the category corresponding to the highest probability in p is the depression state classification result.

[0085] Furthermore, the data augmentation includes: performing synchronous segmentation of each preprocessed modal physiological signal using a non-overlapping sliding window to obtain a basic sample set for each modal physiological signal; then, inputting the basic sample set of each modal physiological signal into a pre-trained class conditional generative adversarial network P-WACGAN to generate a synthetic sample set for each modal physiological signal; and merging the basic sample set and the synthetic sample set to obtain an augmented sample set.

[0086] like Figure 2 As shown, the category-conditional generative adversarial network P-WACGAN includes interconnected generators and discriminators; The generator includes: a first fully connected layer, a first convolutional layer, a first activation function layer, a first normalization layer, a second convolutional layer, a second activation function layer, a second normalization layer, a third convolutional layer, a third activation function layer, a third normalization layer, a fourth convolutional layer, and a fourth activation function layer connected in sequence. The discriminator includes: a fifth convolutional layer, a fifth activation function layer, a first pooling layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a second pooling layer, and a second fully connected layer, connected in sequence.

[0087] The generator is used to generate samples, and the discriminator is used to determine whether the samples generated by the generator are true.

[0088] The first fully connected layer is used to perform preliminary feature mapping on the input random noise vector and the corresponding category condition information, converting the low-dimensional input into a high-dimensional feature representation to meet the input requirements of subsequent convolution operations. The first convolutional layer performs local feature extraction on the mapped features to learn the basic structural features of the time series signal; The first activation function layer introduces nonlinear transformations to improve the network's ability to express complex feature relationships; The first normalization layer is used to normalize the distribution of intermediate features to improve the stability of the model training process.

[0089] The second convolutional layer further extracts high-level features based on the aforementioned features, which is used to enhance the continuity of the generated signal in the time dimension. The second activation function layer is used to enhance the expression of nonlinear features; The second normalization layer normalizes the features, reduces gradient fluctuations, and improves the convergence speed of the model.

[0090] The third convolutional layer is used to reconstruct high-level features, so that the generated signal gradually approximates the distribution features of the real sample in the overall shape. The third activation function layer enhances the model's ability to express complex signal structures; The third normalization layer is used to further stabilize the generation process and improve the consistency and smoothness of the generated signal.

[0091] The fourth convolutional layer maps the high-dimensional features to the target signal dimension, completing the final construction of the generated signal; The fourth activation function layer is used to constrain the output range of the generated signal so that the generated result conforms to the numerical characteristics of the actual physiological signal.

[0092] The fifth convolutional layer is used to perform preliminary feature extraction on the input time-series signal in order to capture the local variation features of the signal; The fifth activation function layer introduces a nonlinear transformation to improve the discriminator's ability to recognize complex signal patterns; The first pooling layer is used to reduce the dimensionality of the features, reducing redundant features while retaining the main discriminative information.

[0093] The sixth and seventh convolutional layers are used to extract the structural features of time-series signals at a higher level to enhance the ability to distinguish differences between different categories of signals; The corresponding sixth and seventh activation function layers are used to enhance the model's nonlinear discrimination capability.

[0094] The eighth and ninth convolutional layers are used to further integrate and compress high-level features, highlighting key information related to authenticity judgment; The corresponding eighth and ninth activation function layers are used to improve the model's responsiveness to complex feature combinations.

[0095] The second pooling layer is used to further reduce feature dimensionality, reduce feature redundancy, and improve the generalization performance of the model. The second fully connected layer is used to comprehensively judge the extracted multi-layer features and output the authenticity judgment result corresponding to the input signal.

[0096] like Figure 2 As shown, the category-conditional generative adversarial network P-WACGAN includes interconnected generators and discriminators, which work together through an adversarial learning mechanism to generate and discriminate time-series physiological signals under category-conditional constraints.

[0097] The generator is used to gradually generate a time-series signal that conforms to the distribution of the target category characteristics under the joint constraints of random noise and category condition information; the discriminator is used to judge the authenticity of the input signal and feed the judgment result back to the generator to guide the generator to continuously optimize the generation effect.

[0098] The data augmentation includes: firstly, using a non-overlapping sliding window to synchronously segment the three modal physiological signals, wherein the window duration of the sliding window is fixed at 5 s and the step size is equal to the window length to ensure that adjacent time windows do not overlap, thereby obtaining a basic training sample set with time window alignment; Then, the class conditional generative adversarial network P-WACGAN is used to generate class-oriented data from the segmented training samples in order to expand the training set and alleviate class imbalance. Among them, the category conditional generative adversarial network P-WACGAN integrates Wasserstein adversarial training and gradient penalty constraints, and introduces an auxiliary classification loss function to achieve controllable generation of depression level categories. At the same time, an adaptive hyperparameter adjustment strategy PBT is introduced during training to improve the stability of adversarial training.

[0099] The category-conditional generative adversarial network includes a generator G and a discriminator / commentator D; wherein the discriminator D has two output branches: an adversarial branch for outputting the "true / false / rating" of a sample, and an auxiliary classification branch for outputting the class probability of a sample.

[0100] The category-conditional generative adversarial network includes a generator G and a discriminator D; the generator G takes a combination of a random noise vector z and a category label y as input and outputs a synthetic sample corresponding to the category. =G(z,y); Discriminator D receives the real sample x and the synthetic sample x. It takes two branches as input and outputs: one is the authenticity score branch D. adv (·), used for the Wasserstein score of the output sample; the second is the auxiliary classification branch D. cls (·) is used to output the sample class probability P(y|x).

[0101] To satisfy the 1-Lipschitz continuity constraint, a gradient penalty term is introduced into the discriminator loss; the real sample x is compared with the generated sample x. Perform linear interpolation to obtain interpolated samples. gradient penalty term Represented as:

[0102] The auxiliary classification loss uses multi-class cross-entropy loss. This is used to constrain the discriminator's ability to model category information and guide the generator to generate samples consistent with the category label y. Represented as:

[0103] The joint loss function of the discriminator D consists of the Wasserstein adversarial loss, the gradient penalty term, and the auxiliary classification loss. The discriminator loss function L D Represented as:

[0104] Where λ gp λ is the gradient penalty coefficient. cls The weighting coefficients are used to assist in the classification loss.

[0105] Loss function of generator G The generator loss function consists of adversarial loss and auxiliary classification loss, aiming to enhance class consistency while improving the realism of generated samples. Represented as:

[0106] During training, the discriminator D and generator G are updated using an alternating iterative approach: in each iteration, the discriminator D is updated first to minimize L. D Then update the generator G to minimize Training stops when the preset number of training rounds is reached or the convergence condition is met, resulting in the trained P-WACGAN generative model.

[0107] The adaptive hyperparameter tuning strategy (PBT) is used to adjust hyperparameters online during training. When a preset evaluation interval is reached, individuals within the population are ranked according to model evaluation metrics. Weaker-performing individuals inherit the network weights (exploit) of better-performing individuals, and their hyperparameters are perturbed (explore). These hyperparameters include at least the learning rate and the gradient penalty coefficient λ. gp To improve the stability of adversarial training and enhance the quality of generated samples, a Population Based Training (PBT) strategy is introduced to schedule hyperparameters online during the training process of the class-conditional generative adversarial network, thereby improving the stability of the generation process and the enhancement effect.

[0108] The trained category conditional generative adversarial network P-WACGAN is used to generate synthetic samples for depression level categories with a small number of samples. The synthetic samples are then merged with real samples to construct an enhanced training set. Targeted amplification is performed for each category until a preset number of categories are reached for training of the subsequent depression state recognition model.

[0109] The beneficial effects of the above technical solution are as follows: In the model training stage, the preprocessed trimodal physiological signals are first synchronously segmented using a non-overlapping sliding window to obtain a basic sample set with time window alignment; and based on the basic sample set, a class conditional generative adversarial network is used to perform targeted generation to expand the training set and alleviate class imbalance. Among them, Wasserstein adversarial training and gradient penalty are used to improve the stability of adversarial training, and an online hyperparameter scheduling strategy is introduced to further improve the stability and enhancement effect of the generation process.

[0110] Furthermore, the total loss function of the model is a weighted combination of cross-entropy loss (CE), Focal loss, mean squared error loss (MSE), and the smoothing term (Smooth).

[0111] The total loss function is a weighted sum of four loss functions, taking into account classification accuracy, sample weights, and prediction probabilities. Its overall expression is as follows:

[0112] in, It is the model's predicted probability for the sample (for positive samples, It is the probability of a positive prediction; for negative samples, (This is the complement of the probability of a negative prediction); λ1, λ2, λ3, and λ4 are the weights of each loss function, used to adjust their influence on the final goal. This represents the Focal Loss function. Represents the cross-entropy loss function. Represents the MSE loss function. This represents the Smooth Loss function.

[0113] Focal Loss focuses on difficult-to-classify samples and reduces the weight of easy samples. By introducing a dynamic scaling factor and a positive-negative sample weight adjustment mechanism, the Focal Loss function effectively solves the problem of imbalance between positive and negative samples in classification tasks, and can significantly improve the detection accuracy and recall of the model.

[0114] The formula for the Focal Loss function is as follows:

[0115] in, It is the model's predicted probability for the sample. It is a parameter that adjusts the weights of positive and negative samples, and is used to adjust the weight ratio between positive and negative samples; It is a parameter that adjusts the weights of easy and difficult samples (also known as the focus parameter).

[0116] The cross-entropy loss function measures the degree of matching between the predicted probabilities and the actual labels. By measuring the difference between the model's predicted probabilities and the true distribution, the cross-entropy loss helps the model gradually adjust its parameters to ultimately achieve optimal classification performance.

[0117] The formula for the cross-entropy loss function is as follows:

[0118] in, It is the model's predicted probability for the sample. It is a sample The true label, if the sample If the true category is the same as the predicted category, then A value of 1 indicates depression, otherwise 0 indicates depression; K represents the classification category (normal, mild, moderate, and severe depression). ).

[0119] Mean Squared Error (MSE): This measures the squared error between the predicted probability and the actual label.

[0120] The MSE loss function is as follows:

[0121] in, It is the model's predicted probability for the sample. Yes, it is a sample The true label, if the sample If the true category is the same as the predicted category, then If it is 1, then it is 0.

[0122] Smooth Loss is used to control gradient smoothing and avoid drastic parameter oscillations.

[0123] The Smooth Loss function is defined as follows:

[0124] in, It is the model's predicted probability for the sample. Yes, it is a sample The true label.

[0125] The depression state recognition network is used to extract features from ECG, PPG and RSP respectively: the three physiological signals are input into the MPARN+Bi-LSTM network to obtain the corresponding modal features, and the modal features are fused at the feature level to obtain fused features; the fused features are input into the Transformer encoder for feature representation learning, and the four depression state recognition results of normal, mild depression, moderate depression and severe depression are output through the fully connected layer.

[0126] The training process includes: sequentially performing downsampling, filtering and denoising, and max-min normalization on the training samples; executing a data augmentation module during the training phase to perform non-overlapping sliding window segmentation and generatively expand the training sample set; then inputting the augmented training samples into the feature extraction module and the feature fusion module, and into the classification and recognition module to output the depression state prediction result; optimizing the model parameters using a total loss function during the training process; stopping training when the total loss function value corresponding to the preset evaluation index no longer decreases in several consecutive iterations, thus obtaining the trained depression state recognition model.

[0127] The classification and recognition module used in this invention employs a Transformer encoder to perform deep semantic representation learning on the fused features, and then sets a classification layer to output a four-class classification result of the depressive state. Specifically, the preprocessed multimodal physiological signals are subjected to feature extraction and feature fusion to obtain fused features. The fused features are then input into a Transformer encoder to obtain semantic feature representations. These semantic feature representations are then input into a fully connected classification layer to obtain logits for each category. Finally, the predicted probabilities of normal state, mild depressive state, moderate depressive state, and severe depressive state are output through a Softmax function, and the category corresponding to the highest predicted probability is used as the recognition result.

[0128] During the training phase, training samples labeled with depressive states are input into the depression state recognition model. The model performs deep semantic representation learning on the fused features using a Transformer encoder and outputs predicted probabilities for four types of depressive states through a classification layer. During training, the model parameters are iteratively optimized based on a classification loss function, and a validation set is created to monitor the model's generalization performance. Training stops when the validation set loss no longer decreases within a preset number of rounds, and the model parameters corresponding to the minimum validation set loss are retained, resulting in the trained depression state recognition model. During the inference phase, the sample to be identified is input into the trained depression state recognition model. The model outputs predicted probabilities for normal, mild, moderate, and severe depressive states through a classification layer, and the category corresponding to the highest predicted probability is used as the depression state recognition result.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data-augmented multimodal physiological signal-based system for identifying depressive states, characterized by: include: The acquisition module is configured to acquire multimodal physiological signal data to be identified. The preprocessing module is configured to preprocess the physiological signal data of each modality to obtain preprocessed physiological signal samples of each modality. The identification module is configured to: input multimodal physiological signal samples into a trained depression state identification model to obtain a depression state identification result; wherein, the trained depression state identification model is used to extract features from each preprocessed modality of physiological signal to obtain features corresponding to each modality of physiological signal; fuse the features corresponding to the multimodal physiological signals to obtain fused features; and classify the fused features to obtain a depression state identification result. In the training process of the depression state recognition model, a training set is used for training. The training set includes multimodal physiological signal data with known depression state labels. During the training process, the training set is first preprocessed, and then the preprocessed results are augmented with data. Next, the augmented data is used to train the model. When the total loss function value of the model no longer decreases, the training is stopped, and the trained depression state recognition model is obtained.

2. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 1, characterized in that, The preprocessing of each modality of physiological signal data to obtain preprocessed samples of each modality of physiological signal includes: downsampling, filtering and normalization. The downsampling process is used to uniformly reduce the sampling rate of ECG, PPG and RSP signals from the original 1000Hz to a preset sampling rate of 256Hz. The filtering process includes: filtering the electrocardiogram signal, filtering the pulse signal, and filtering the respiration signal; The filtering process for the electrocardiogram (ECG) signal includes: using the sym8 wavelet basis for discrete wavelet decomposition and reconstruction, with the number of decomposition layers fixed at 8, and using the VisuShrink threshold and soft threshold functions to perform threshold denoising on the coefficients to obtain the filtered ECG signal. The filtering process for the pulse signal includes: using the db4 wavelet basis for discrete wavelet decomposition and reconstruction, with the number of decomposition layers fixed at 5, and using the VisuShrink threshold and soft threshold functions to perform threshold denoising on the coefficients before reconstruction to obtain the filtered pulse signal. The filtering process for the respiratory signal includes: using the db8 wavelet basis for discrete wavelet decomposition and reconstruction, with the number of decomposition layers fixed at 8, and using the VisuShrink threshold and soft threshold functions to perform threshold denoising on the coefficients before reconstruction to obtain the filtered respiratory signal. The normalization process uses max-min normalization to scale the amplitude of the filtered signal.

3. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 1, characterized in that, The trained depression state recognition model includes: a first feature extraction unit, a second feature extraction unit, and a third feature extraction unit; The first feature extraction unit is used to input the preprocessed electrocardiogram (ECG) signal, extract features from the ECG signal to obtain ECG features, and input the extracted ECG features into the fusion unit; The second feature extraction unit is used to input the preprocessed pulse signal, extract features from the pulse signal to obtain pulse features, and input the extracted pulse features into the fusion unit; The third feature extraction unit is used to input the preprocessed respiratory signal, extract features from the respiratory signal to obtain respiratory features, and input the extracted respiratory features into the fusion unit; The fusion unit fuses the three input features to obtain fused features; The classifier classifies the fused features to obtain the classification result.

4. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 3, characterized in that, The internal structures of the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit are identical. The first feature extraction unit includes a multipath adaptive residual network MPARN and a Bi-LSTM network connected in sequence. The classifier includes a Transformer encoder and a classification output layer connected in sequence. The classification output layer includes a global pooling layer and a fully connected layer connected in sequence. The depression state classification results include normal, mild depression, moderate depression, and severe depression.

5. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 1, characterized in that, The data augmentation includes: performing synchronous segmentation of each preprocessed modal physiological signal using a non-overlapping sliding window to obtain a basic sample set for each modal physiological signal; then, inputting the basic sample set of each modal physiological signal into a pre-trained class conditional generative adversarial network P-WACGAN to generate a synthetic sample set for each modal physiological signal; and merging the basic sample set and the synthetic sample set to obtain an augmented sample set.

6. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 5, characterized in that, The category conditional generative adversarial network P-WACGAN includes: a generator and a discriminator connected to each other; the generator includes: a first fully connected layer, a first convolutional layer, a first activation function layer, a first normalization layer, a second convolutional layer, a second activation function layer, a second normalization layer, a third convolutional layer, a third activation function layer, a third normalization layer, a fourth convolutional layer, and a fourth activation function layer connected in sequence. The discriminator includes: a fifth convolutional layer, a fifth activation function layer, a first pooling layer, a sixth convolutional layer, a sixth activation function layer, a seventh convolutional layer, a seventh activation function layer, an eighth convolutional layer, an eighth activation function layer, a ninth convolutional layer, a ninth activation function layer, a second pooling layer, and a second fully connected layer, all connected in sequence; the generator is used to generate samples, and the discriminator is used to determine whether the samples generated by the generator are genuine.

7. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 1, characterized in that, The data augmentation includes: firstly, using a non-overlapping sliding window to synchronously segment the three modal physiological signals, wherein the window duration of the sliding window is fixed at 5 s and the step size is equal to the window length to ensure that adjacent time windows do not overlap, thereby obtaining a basic training sample set with time window alignment; Then, the class conditional generative adversarial network P-WACGAN is used to generate class-oriented data from the segmented training samples in order to expand the training set and alleviate class imbalance. Among them, the category conditional generative adversarial network P-WACGAN integrates Wasserstein adversarial training and gradient penalty constraints, and introduces an auxiliary classification loss function to achieve controllable generation of depression level categories. At the same time, an adaptive hyperparameter adjustment strategy PBT is introduced during training to improve the stability of adversarial training.

8. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 7, characterized in that, The category-conditional generative adversarial network includes a generator G and a discriminator / commenter D; wherein the discriminator D has two output branches: an adversarial branch for outputting the "true / false / rating" of a sample, and an auxiliary classification branch for outputting the class probability of a sample; The category-conditional generative adversarial network includes a generator G and a discriminator D; the generator G takes a combination of a random noise vector z and a category label y as input and outputs a synthetic sample corresponding to the category. =G(z,y); Discriminator D receives the real sample x and the synthetic sample x. It takes two branches as input and outputs: one is the authenticity score branch D. adv (·), used for the Wasserstein score of the output sample; the second is the auxiliary classification branch D. cls (·) is used to output the sample class probability P(y|x).

9. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 8, characterized in that, A gradient penalty term is introduced into the discriminator loss; the real sample x is compared with the generated sample. Perform linear interpolation to obtain interpolated samples. gradient penalty term Represented as: ； The auxiliary classification loss uses multi-class cross-entropy loss. This is used to constrain the discriminator's ability to model category information and guide the generator to generate samples consistent with the category label y. Represented as: ； The joint loss function of the discriminator D consists of the Wasserstein adversarial loss, the gradient penalty term, and the auxiliary classification loss. The discriminator loss function L D Represented as: ； Where λ gp λ is the gradient penalty coefficient. cls Weighting coefficients for auxiliary classification loss; Loss function of generator G The generator loss function consists of adversarial loss and auxiliary classification loss, aiming to enhance class consistency while improving the realism of generated samples. Represented as: ； During training, the discriminator D and generator G are updated using an alternating iterative approach: in each iteration, the discriminator D is updated first to minimize L. D Then update the generator G to minimize ; Training stops when the preset number of training rounds is reached or the convergence condition is met, resulting in the trained P-WACGAN generative model.

10. The data-enhanced multimodal physiological signal-based depressive state recognition system as described in claim 1, characterized in that, The total loss function of the model is a weighted combination of cross-entropy loss (CE), Focal loss, mean squared error loss (MSE), and smoothing term (Smooth). The total loss function is a weighted sum of four loss functions, taking into account classification accuracy, sample weights, and prediction probabilities. The overall expression is as follows: ； in, λ1, λ2, λ3, and λ4 are the predicted probabilities of the model for the samples; λ1, λ2, λ3, and λ4 are the weights of each loss function, used to adjust their influence on the final goal. This represents the Focal Loss function. Represents the cross-entropy loss function. Represents the MSE loss function. This represents the Smooth Loss function.

Images