Stress identification method, system, medium and device based on multi-mode signal fusion

Abstract

The present invention relates to the technical field of tension recognition, and discloses a tension recognition method, system, medium and device based on multi-mode signal fusion, including: obtaining physiological signals; based on the physiological signals, obtaining time series features based on time domain extraction, time series features based on frequency domain extraction and time series features based on statistical domain through shallow feature extraction methods in time domain, frequency domain and statistical domain, and then extracting multi-scale deep features through a multi-scale deep feature extraction method; for the multi-scale deep features, calculating the correlation between deep features of different scales, deleting deep features with correlation exceeding a threshold, and fusing the deep features using a weighted summation method to obtain multi-scale fusion features; based on the multi-scale fusion features, obtaining a classification result of tension. The accuracy and objectivity of tension assessment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of tension recognition, and in particular to a tension recognition method, system, medium and device based on multi-mode signal fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Current stress assessment techniques mostly rely on traditional psychological scales, which have problems such as difficulty in administration, strong subjectivity, time-consuming and labor-intensive, limited scope of application, and poor universality.

[0004] Currently, stress assessment mostly uses scales introduced from abroad, such as the Self-Rating Stress Scale (STS), the State-Trait Stress Inventory (STAI) and the GAD (Generalized Anxiety Disorder)-7 Psychological Stress Screening Scale. This assessment method is time-consuming and labor-intensive, has strong subjectivity and unstable results, and is also highly culturally sensitive, which makes the assessment results less accurate and has a very limited scope of application.

[0005] At present, most tension recognition methods use a single signal, the physiological information mined is limited, and the recognition performance is poor; although there are a few tension recognition methods based on fusion features, the current methods do not pay attention to the influence of different features on tension recognition, and the accuracy and objectivity of tension assessment are poor. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a tension recognition method, system, medium and device based on multi-mode signal fusion, which extracts shallow features of a number of physiological signals in the time domain, frequency domain and statistical domain, extracts multi-scale deep features, calculates the correlation between the multi-scale features, deletes the multi-scale features whose correlation exceeds a threshold, retains the multi-scale features with large correlation differences, and dynamically adjusts the importance of features of different scales, so that the multi-scale fusion features can pay more attention to features of certain scales, realizes efficient fusion of the physiological signals of the subjects, and improves the accuracy and objectivity of tension assessment.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a stress level identification method based on multi-modal signal fusion, which comprises:

[0009] Acquire physiological signals;

[0010] Based on the physiological signal, after obtaining time series features extracted based on time domain, time series features extracted based on frequency domain and time series features extracted based on statistical domain through shallow feature extraction methods in time domain, frequency domain and statistical domain, multi-scale deep features are extracted through multi-scale deep feature extraction method;

[0011] For the multi-scale deep features, the correlation between deep features of different scales is calculated, and after deleting the deep features whose correlation exceeds a threshold, the deep features are fused by a weighted summation method to obtain multi-scale fused features;

[0012] Based on the multi-scale fusion features, a classification result of tension is obtained.

[0013] Furthermore, the physiological signals include brain waves, body temperature, blood oxygen, respiration, heart rate and skin electricity.

[0014] Furthermore, the time series features extracted based on the time domain include: autocorrelation, centroid, difference mean, difference absolute value mean, difference median, difference absolute value median, difference absolute value sum, entropy, peak-to-trough distance, curve coverage area, maximum number of peaks, minimum number of peaks and zero crossing rate.

[0015] Furthermore, the time series features extracted based on frequency domain include: fast Fourier transform, Fourier transform average coefficient, wavelet transform, wavelet absolute mean, wavelet standard deviation, wavelet variance, spectral distance, spectrum fundamental frequency, spectrum maximum frequency, spectrum intermediate frequency and spectrum maximum peak.

[0016] Furthermore, the time series features based on the statistical domain include: maximum value, minimum value, mean, median, skewness, kurtosis, histogram, interquartile range, absolute error mean, absolute error median, root mean square, standard deviation, variance, empirical distribution function percentile and empirical distribution function slope.

[0017] Furthermore, it also includes: extracting multi-scale deep features after unifying the dimensions of the time series features extracted based on time domain, the time series features extracted based on frequency domain and the time series features based on statistical domain based on interval quantization and representation learning.

[0018] Furthermore, the correlation is calculated by matrix multiplication after the deep features are expressed in matrix form.

[0019] A second aspect of the present invention provides a stress level identification system based on multi-modal signal fusion, comprising:

[0020] A signal acquisition module, which is configured to: acquire physiological signals;

[0021] A feature extraction module is configured to: based on the physiological signal, after obtaining time series features extracted based on time domain, time series features extracted based on frequency domain and time series features extracted based on statistical domain by shallow feature extraction methods in time domain, frequency domain and statistical domain, extract multi-scale deep features by multi-scale deep feature extraction method;

[0022] A feature fusion module is configured to: for the multi-scale deep features, calculate the correlation between deep features of different scales, delete the deep features whose correlation exceeds a threshold, and fuse the deep features using a weighted summation method to obtain a multi-scale fusion feature;

[0023] The classification module is configured to obtain a classification result of tension based on the multi-scale fusion features.

[0024] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor and implements the steps of the tension identification method based on multimodal signal fusion as described above when the program is executed by the processor.

[0025] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the method for identifying tension based on multimodal signal fusion as described above are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a tension recognition method based on multi-mode signal fusion, which extracts shallow features of several physiological signals in time domain, frequency domain and statistical domain, and then extracts multi-scale deep features, calculates the correlation between the multi-scale deep features by matrix multiplication, deletes the deep features whose correlation exceeds a threshold, retains the deep features with large correlation differences, and dynamically adjusts the importance of deep features of different scales, so that the multi-scale fusion features can pay more attention to the deep features of certain scales, realizes efficient fusion of the physiological signals of the subjects, and improves the accuracy and objectivity of tension assessment.

[0028] The present invention provides a tension recognition method based on multi-mode signal fusion, which saves time and labor, improves evaluation efficiency, and is also applicable to multiple scenarios and multiple groups of people. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0030] Figure 1This is a flow chart of a method for identifying stress based on multi-modal signal fusion according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0033] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention is further described below with reference to the accompanying drawings and embodiments.

[0034] Embodiment 1

[0035] The purpose of this embodiment is to provide a method for identifying stress based on multi-mode signal fusion. Figure 1 As shown, the following steps are included:

[0036] Step 1: The wearable physiological signals of the subjects are collected in real time.

[0037] The experimental paradigm for real-time collection of wearable physiological signals includes at least virtual reality scenarios such as high-altitude work and speech competitions. Among them, the virtual scene of high-altitude work can be designed as the gradual breaking of multiple ropes during rope suspension work, so as to induce and increase the tension of the subjects.

[0038] The collected wearable physiological signals include at least EEG, body temperature, blood oxygen, respiration, heart rate and skin electricity, etc. The collection method is to use a wireless and portable EEG belt, wearable bracelet, smart chest belt and infrared finger clip to collect in real time, and save the collected wearable physiological signals in a multimodal queue database.

[0039] Step 2: Based on the wearable physiological signals, shallow features (including: time series features extracted based on time domain, time series features extracted based on frequency domain, and time series features based on statistical domain) are obtained through shallow feature extraction methods in time domain, frequency domain, and statistical domain.

[0040] Shallow feature extraction methods based on time domain, frequency domain and statistical domain at least include feature extraction methods in time domain, frequency domain and statistical domain.

[0041] Among them, the time series features extracted based on the time domain include at least: autocorrelation, centroid, difference mean, difference absolute value mean, difference median, difference absolute value median, difference absolute value sum, entropy, peak-to-trough distance, curve coverage area, maximum number of peaks, minimum number of peaks, zero crossing rate, etc.

[0042] Among them, the time series features extracted based on frequency domain include at least: fast Fourier transform, Fourier transform average coefficient, wavelet transform, wavelet absolute mean, wavelet standard deviation, wavelet variance, spectral distance, spectrum fundamental frequency, spectrum maximum frequency, spectrum intermediate frequency, spectrum maximum peak value, etc.

[0043] Among them, the time series features based on the statistical domain include at least: maximum value, minimum value, mean, median, skewness, kurtosis, histogram, interquartile range, absolute error mean, absolute error median, root mean square, standard deviation, variance, empirical distribution function percentile and empirical distribution function slope, etc.

[0044] Step 3: Based on shallow features, a fused unified-dimensional multimodal physiological feature is obtained through a heterogeneous data dimension unification and multimodal feature fusion method based on representation learning.

[0045] The heterogeneous data dimension unification and multi-modal feature fusion methods based on representation learning include at least the following two methods:

[0046] (1) A dimensional unification method based on interval quantization, which can convert one-dimensional data into two-dimensional data.

[0047] The dimension unification method based on interval quantization can be assisted by methods such as the time window method or the sliding window method. For example, the time window method: divide the one-dimensional data into non-overlapping or partially overlapping time windows, and apply interval quantization in each time window. In this way, the data in each time window can be regarded as a new one-dimensional data, and the new one-dimensional data is stacked, thereby realizing the transformation of a one-dimensional data into a two-dimensional data containing multiple new one-dimensional data, and realizing the purpose of multi-dimensional data sharing the same dimensional space.

[0048] For example, if the one-dimensional data is [1,2,3,…,4,5,…,64], the converted two-dimensional data is [1,…,8;9,10,…,16;17,18,…,24;…;57,58,…,64].

[0049] (2) A dimensionality unification method based on representation learning, which can map heterogeneous data into a low-dimensional embedding space.

[0050] The dimensionality unification method based on representation learning aims to ensure similar structures in different dimensions by learning the representation of the data. The goal of representation learning is to find a mapping that maps the input data to a more structured and interpretable representation space. Specifically, it may include: data preparation, building a representation learning model (autoencoder, variational autoencoder, adversarial generative network, etc.), defining a loss function (designing a loss function that can quantify the similarity between the original data and the learned representation, which helps to ensure that the learned representation is consistent in different dimensions), training the model, evaluating the learned representation, adjusting hyperparameters, and applying the learned representation. Then, high-level information is aligned in the embedding space to achieve dimensionality unification of heterogeneous data, and then the multimodal physiological features of the unified dimension are fused to facilitate the extraction of multi-scale deep features in the next step.

[0051] Among them, heterogeneous data mainly refers to a variety of data with uneven data quality. Heterogeneous data mapping refers to the process of mapping data from different sources, formats or structures into a unified and understandable format or structure when processing them. This mapping is usually to facilitate data integration, analysis and utilization. The dimension unification method based on interval quantization and the dimension unification method based on representation learning are both methods for unifying the dimensions of heterogeneous data. Heterogeneous data can only be processed together after the dimensions are unified. Otherwise, the formats are different and the dimensions are incompatible, and unified processing cannot be achieved, or unified processing is difficult.

[0052] Step 4: Based on the fused multi-modal physiological features of unified dimension, the multi-scale deep features of wearable physiological signals are extracted through a multi-scale deep feature extraction method based on convolutional neural network (CNN).

[0053] The multi-scale deep feature extraction method based on CNN adopts a convolutional neural network consisting of an input layer, three convolutional layers, three pooling layers and three activation function layers.

[0054] Among them, the multi-scale deep features are matrices or vectors, which can represent the multi-scale deep features of wearable physiological signals.

[0055] Step 5: Multi-scale feature fusion and RNN-based recurrent neural network classification method. Multi-scale deep features are efficiently fused and the classification results of tension are obtained.

[0056] Multi-scale feature fusion and RNN-based recurrent neural network classification methods can efficiently fuse multi-scale deep features. Specifically, for multi-scale deep features, the correlation between deep features of different scales is calculated, and after deleting the deep features whose correlation exceeds the threshold, the weighted summation method is used to fuse the deep features to obtain multi-scale fusion features.

[0057] First, the correlation between deep features of different scales is calculated by matrix multiplication:

[0058] Step 1: Matrix representation of multi-scale deep features: Represent the multi-scale deep features in matrix form, where each row corresponds to a sample and each column corresponds to a deep feature of one scale. Suppose there are two scales of deep feature matrices, namely, deep feature matrices X and the deep feature matrix Y , where the matrix X yes m × n The matrix represents m samples and n deep features, matrix Y yes m × p The matrix represents m samples and p A deep feature.

[0059] Step 2: Calculate the covariance matrix: Calculate the matrix X and matrix Y The covariance matrix of The first ( i , j ) elements represent the matrix X No. i Deep features and matrices Y No. j The covariance between deep features, where Representation Matrix X The transpose of .

[0060] Step 3: Calculate the correlation coefficient matrix: Calculate the correlation coefficient matrix using the covariance matrix , the first ( i , j ) elements represent the matrix X No. i Deep features and matrices Y No. j The correlation coefficient between deep features is: and is a matrix X and matrix Y The autocovariance matrix of .

[0061] In this embodiment, the correlation calculation is performed using matrix multiplication: the calculation matrix X Each deep feature in the matrix Y The correlation between each deep feature in , we can use matrix multiplication: ,in, Representation Matrix X The transpose of express The square root of the inverse matrix of .

[0062] Secondly, the multi-scale deep features whose correlation exceeds the threshold are deleted (if the correlation of deep features of two scales exceeds the threshold, you can choose to delete any one or both of them), retain the deep features with large correlation differences, and then efficiently fuse the retained deep features to output multi-scale fused features.

[0063] The specific fusion method for fusing deep features uses the weighted summation method ,in, N represents the number of deep features, Represents the fused deep features (multi-scale fusion features), Represents the adaptive training of the model using the self-attention mechanism weight learning k The weight of deep features, Indicates k The self-attention mechanism weight learning model can dynamically adjust the importance of deep features of different scales, so that the multi-scale fusion features can pay more attention to the deep features of certain scales.

[0064] Finally, the multi-scale fusion features are input into the RNN (recurrent neural network)-based classification method, and the final classification results of mild, moderate or severe tension are output, which improves the classification performance of tension assessment.

[0065] The stress level identification method based on multi-modal signal fusion provided in this embodiment uses a multi-modal physiological data fusion algorithm to efficiently fuse the physiological signals of the subject, quantify the perceived stress level of the subject, and improve the accuracy and objectivity of stress level assessment.

[0066] The tension recognition method based on multi-modal signal fusion provided in this embodiment extracts shallow features of several physiological signals in the time domain, frequency domain and statistical domain, and then extracts multi-scale deep features, calculates the correlation between the multi-scale deep features by matrix multiplication, deletes the deep features whose correlation exceeds the threshold, retains the deep features with large correlation differences, and dynamically adjusts the importance of deep features of different scales, so that the multi-scale fusion features can pay more attention to the features of certain deep scales, realizes efficient fusion of the physiological signals of the subjects, and improves the accuracy and objectivity of tension assessment.

[0067] Embodiment 2

[0068] The purpose of this second embodiment is to provide a stress level recognition system based on multi-mode signal fusion.

[0069] A signal acquisition module, which is configured to: acquire physiological signals;

[0070] A feature extraction module is configured to: based on the physiological signal, after obtaining time series features extracted based on time domain, time series features extracted based on frequency domain and time series features extracted based on statistical domain by shallow feature extraction methods in time domain, frequency domain and statistical domain, extract multi-scale deep features by multi-scale deep feature extraction method;

[0071] A feature fusion module is configured to: for the multi-scale deep features, calculate the correlation between deep features of different scales, delete the deep features whose correlation exceeds a threshold, and fuse the deep features using a weighted summation method to obtain a multi-scale fusion feature;

[0072] The classification module is configured to obtain a classification result of tension based on the multi-scale fusion features.

[0073] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0074] Embodiment 3

[0075] This embodiment provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor. When the program is executed by the processor, the steps in the method for identifying tension based on multimodal signal fusion as described in the first embodiment are implemented.

[0076] Embodiment 4

[0077] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps in the method for identifying stress based on multimodal signal fusion as described in the first embodiment are implemented.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0079] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for identifying tension based on multi-modal signal fusion, characterized in that: include: Acquire physiological signals; Based on the physiological signal, after obtaining time series features extracted based on time domain, time series features extracted based on frequency domain and time series features extracted based on statistical domain through shallow feature extraction methods in time domain, frequency domain and statistical domain, multi-scale deep features are extracted through multi-scale deep feature extraction method; For the multi-scale deep features, the correlation between deep features of different scales is calculated, and after deleting the deep features whose correlation exceeds a threshold, the deep features are fused by a weighted summation method to obtain multi-scale fused features; Based on the multi-scale fusion features, a classification result of tension is obtained; The correlation between deep features of different scales is calculated by matrix multiplication: the multi-scale deep features are represented as a matrix, where each row corresponds to a sample and each column corresponds to a deep feature of one scale. For each deep feature matrix with m samples, X and the deep feature matrix Y , calculate the matrix X and matrix Y The covariance matrix of , using the covariance matrix to calculate the correlation coefficient matrix , Representation Matrix X Each deep feature in the matrix Y The correlation between each deep feature in ; The multi-scale deep features whose correlation exceeds the threshold are deleted. If the correlation of the deep features of two scales exceeds the threshold, you can choose to delete any one or both of them, and retain the deep features with large correlation differences. The retained deep features are efficiently fused to output multi-scale fused features; The specific fusion method for fusing deep features uses the weighted summation method ,in, N represents the number of deep features, represents the deep features of the fusion, Represents the adaptive training of the model using the self-attention mechanism weight learning k The weight of deep features, Indicates k The self-attention mechanism weight learning model dynamically adjusts the importance of deep features of different scales, making the multi-scale fusion features pay more attention to the deep features of certain scales.

2. The method for identifying stress based on multimodal signal fusion according to claim 1, characterized in that: The physiological signals include brain waves, body temperature, blood oxygen, respiration, heart rate and skin electricity.

3. The method for identifying stress based on multi-mode signal fusion according to claim 1, characterized in that: The time series features extracted based on the time domain include: autocorrelation, centroid, difference mean, difference absolute value mean, difference median, difference absolute value median, difference absolute value sum, entropy, peak-to-trough distance, curve coverage area, maximum peak number, minimum peak number and zero crossing rate.

4. The method for identifying stress based on multi-mode signal fusion according to claim 1, characterized in that: The time series features extracted based on frequency domain include: fast Fourier transform, Fourier transform average coefficient, wavelet transform, wavelet absolute mean, wavelet standard deviation, wavelet variance, spectral distance, spectrum fundamental frequency, spectrum maximum frequency, spectrum intermediate frequency and spectrum maximum peak.

5. The method for identifying stress based on multi-mode signal fusion according to claim 1, characterized in that: The time series features based on the statistical domain include: maximum value, minimum value, mean, median, skewness, kurtosis, histogram, interquartile range, absolute error mean, absolute error median, root mean square, standard deviation, variance, empirical distribution function percentile and empirical distribution function slope.

6. The method for identifying stress based on multi-mode signal fusion according to claim 1, characterized in that: Also includes: After the time series features extracted based on time domain, the time series features extracted based on frequency domain and the time series features based on statistical domain are unified in dimension based on interval quantization and representation learning, multi-scale deep features are extracted.

7. The tension recognition system based on multi-modal signal fusion is characterized by: include: A signal acquisition module, which is configured to: acquire physiological signals; A feature extraction module is configured to: based on the physiological signal, after obtaining time series features extracted based on time domain, time series features extracted based on frequency domain and time series features extracted based on statistical domain by shallow feature extraction methods in time domain, frequency domain and statistical domain, extract multi-scale deep features by multi-scale deep feature extraction method; A feature fusion module is configured to: for the multi-scale deep features, calculate the correlation between deep features of different scales, delete the deep features whose correlation exceeds a threshold, and fuse the deep features using a weighted summation method to obtain a multi-scale fusion feature; A classification module, configured to: obtain a classification result of tension based on the multi-scale fusion features; The correlation between deep features of different scales is calculated by matrix multiplication: the multi-scale deep features are represented as a matrix, where each row corresponds to a sample and each column corresponds to a deep feature of one scale. For each deep feature matrix with m samples, X and the deep feature matrix Y , calculate the matrix X and matrix Y The covariance matrix of , using the covariance matrix to calculate the correlation coefficient matrix , Representation Matrix X Each deep feature in the matrix Y The correlation between each deep feature in ; The multi-scale deep features whose correlation exceeds the threshold are deleted. If the correlation of the deep features of two scales exceeds the threshold, you can choose to delete any one or both of them, and retain the deep features with large correlation differences. The retained deep features are efficiently fused to output multi-scale fused features; The specific fusion method for fusing deep features uses the weighted summation method ,in, N represents the number of deep features, represents the deep features of the fusion, Represents the adaptive training of the model using the self-attention mechanism weight learning k The weight of deep features, Indicates k The self-attention mechanism weight learning model dynamically adjusts the importance of deep features of different scales, making the multi-scale fusion features pay more attention to the deep features of certain scales.

8. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor, characterized in that: When the program is executed by a processor, the steps in the stress level identification method based on multi-modal signal fusion as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the stress level identification method based on multimodal signal fusion as described in any one of claims 1 to 6 are implemented.

Images