Non-contact stress detection method and system based on remote pulse wave signal

Abstract

The application provides a non-contact stress detection method and system based on a remote pulse wave signal, and relates to the technical field of stress detection. The technical scheme provided by the application combines rPPG signals and first derivative signals thereof as a non-contact stress detection basis from the perspectives of physiological signal timing change characteristics and instantaneous change characteristics, automatically extracts features related to stress states by using a CNN+LSTM network, makes different signal features complementary to each other, and has stronger robustness to signal noise compared with a method of manually calculating and extracting heart rate variability features. In addition, the above processing process does not require the participation of professional technicians, can reduce the professional requirements for operators, and improves the application range of the non-contact stress detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of stress detection, and particularly relates to a non-contact stress detection method and system based on a remote pulse wave signal. BACKGROUND

[0002] In the existing stress state recognition method based on physiological signals, a contact type device such as an electrocardiograph or a bracelet is often used to obtain physiological signals of a user. However, in the signal acquisition process, the contact type device will cause discomfort when being attached to the skin, and long-term use will cause deformation of blood vessels and other tissues at the contact site, thereby affecting the monitoring result of the physiological signals of the user. In addition, some people's skin may have an allergic reaction to the base material of the contact type device. Therefore, the remote photoplethysmography (rPPG) signal has gradually become a commonly used physiological signal in the existing stress state recognition method.

[0003] In the existing non-contact stress detection method based on the rPPG signal, signal peaks in the rPPG signal are obtained, and the peak interval (i.e., the R-R interval) is further calculated, so as to extract a heart rate variability (HRV) feature. The HRV feature is a set of artificially set features for the R-R interval, and is often extracted by manual calculation in the prior art. The above feature extraction method not only requires the user to have higher professional knowledge and tuning experience in the field of stress detection, but also is difficult to capture the complexity and diversity of the data in the feature extraction process.

[0004] Based on the above factors, the existing non-contact stress detection method based on the rPPG signal needs manual intervention and relies too much on the professionalism of the operator, so that the detection efficiency is low. SUMMARY

[0005] (I) Technical problems solved

[0006] In view of the deficiencies in the prior art, the present application provides a non-contact stress detection method and system based on a remote pulse wave signal, which solves the technical problems of the existing non-contact stress detection method based on the rPPG signal, which needs manual intervention, relies too much on the professionalism of the operator, has low detection efficiency, and has low accuracy.

[0007] (II) Technical solutions

[0008] To achieve the above object, the present application is implemented by the following technical solutions:

[0009] In a first aspect of the present application, a non-contact stress detection method based on a remote pulse wave signal is provided, the method comprising:

[0010] acquire a remote pulse wave signal of a preset time period as a first signal;

[0011] acquire a first derivative of the first signal as a second signal;

[0012] input the first signal and the second signal into a preset stress detection network, and acquire a binary classification result output by the stress detection network as a stress detection result; wherein the stress detection network comprises a convolution layer, a pooling layer, a wave peak attention module, a long short-term memory network, and a fully connected layer; the convolution layer and the pooling layer are used for signal feature coding, the wave peak attention module is used for introducing a wave peak detection auxiliary task and calculating time attention weights according to wave peak positions, the long short-term memory network is used for extracting time sequence information, and the fully connected layer is used for acquiring a stress state recognition result.

[0013] Optionally, the processing process of the preset stress detection network on the first signal and the second signal comprises:

[0014] after the first signal and the second signal are input, signal features of the first signal are extracted based on a preset first convolution layer and a first pooling layer as a first feature map, and signal features of the second signal are extracted based on a preset second convolution layer and a second pooling layer as a second feature map;

[0015] signal features of the first signal are extracted based on a preset third convolution layer as a third feature map, and signal features of the third feature map are extracted based on a first wave peak attention module as a fourth feature map;

[0016] signal features of the second signal are extracted based on a preset fourth convolution layer as a fifth feature map, and signal features of the fifth feature map are extracted based on a second wave peak attention module as a sixth feature map; wherein the fourth convolution layer has the same structure as the third convolution layer; the first wave peak attention module has the same structure as the second wave peak attention module and shares parameters with the second wave peak attention module;

[0017] the fourth feature map is subjected to feature extraction based on a preset fifth convolution layer and a fifth pooling layer to obtain a seventh feature map, and the seventh feature map and the first feature map are subjected to feature fusion to obtain an eighth feature map;

[0018] the sixth feature map is subjected to feature extraction based on a preset sixth convolution layer and a sixth pooling layer to obtain a ninth feature map, and the ninth feature map and the second feature map are subjected to feature fusion to obtain a tenth feature map;

[0019] perform feature extraction on the eighth feature map based on a preset seventh convolutional layer and a seventh pooling layer to obtain an eleventh feature map; and perform feature extraction on the tenth feature map based on an eighth convolutional layer and an eighth pooling layer to obtain a twelfth feature map; wherein the first convolutional layer and the second convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, and the eighth convolutional layer have the same structure; and the first pooling layer and the second pooling layer, the fifth pooling layer, the sixth pooling layer, the seventh pooling layer, and the eighth pooling layer have the same structure;

[0020] The eleventh feature map and the twelfth feature map are channel spliced, and the feature map after channel splicing is input into a classifier, so that the classifier outputs a binary classification result of stress state recognition; wherein the classifier is composed of a long short-term memory network and a fully connected network.

[0021] Optionally, the peak attention module includes a peak detection branch, a max-pooling branch, a weight adjustment branch, and a residual attention branch; wherein the peak attention module includes a first peak attention module and a second peak attention module.

[0022] The processing process of the peak attention module on the input feature map includes:

[0023] The peak detection branch is used to obtain a peak detection result corresponding to the feature map; wherein the feature map includes a third feature map and a fifth feature map.

[0024] The max-pooling branch is used to obtain a global pooling weight coefficient of the feature map.

[0025] The weight adjustment branch is used to obtain a weight adjustment coefficient of the feature map.

[0026] The peak detection result, the global pooling weight coefficient, and the weight adjustment coefficient are fused to obtain a comprehensive weight coefficient.

[0027] The SoftMax function is used to map process the comprehensive weight coefficient to obtain a time attention weight.

[0028] The attention weight is multiplied with the feature map in the time domain to obtain a feature map with enhanced time importance.

[0029] The feature map with enhanced time importance is fused with the feature map to obtain a fourth feature map or a sixth feature map.

[0030] Optionally, the process of obtaining the peak detection result corresponding to the feature map based on the peak detection branch includes:

[0031] The feature map is subjected to large-scale receptive field feature extraction based on a double-layer one-dimensional convolution layer and a LeakyReLU function to obtain first to-be-processed wave peak features, and subjected to small-scale receptive field feature extraction based on a single-layer one-dimensional convolution layer and a LeakyReLU function to obtain second to-be-processed wave peak features;

[0032] The first to-be-processed wave peak features and the second to-be-processed wave peak features are subjected to addition processing to obtain third to-be-processed wave peak features containing multi-scale receptive fields;

[0033] The third to-be-processed wave peak features are subjected to wave peak detection calculation based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array of the same length as the input signal as a wave peak detection result.

[0034] Optionally, the process of obtaining the global pooling weight coefficient of the feature map based on the maximum pooling branch comprises:

[0035] The feature map is subjected to global pooling weight calculation based on a global maximum pooling layer and a Sigmoid function to obtain a one-dimensional array of the same length as the input signal, that is, a global pooling weight coefficient.

[0036] Optionally, the process of obtaining the weight adjustment coefficient of the feature map based on the weight adjustment branch comprises:

[0037] The feature map is subjected to adjustment weight calculation based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array of the same length as the input signal, that is, a weight adjustment coefficient.

[0038] Optionally, the loss function of the preset stress detection network is:

[0039] wherein, Loss stress represents a loss value of stress state determination, N represents a total number of samples, i represents an i-th sample, represents a binary classification result of the stress detection network corresponding to the sample i, represents a stress state label corresponding to the sample i.

[0040] represents a wave peak detection loss value of the first signal, represents a wave peak detection loss value of the second signal, and wherein, Loss peak includes or L represents a total number of signal sampling points of a single sample, and j represents a j-th sampling point of the signal. a peak label representing a jth sampling point of an ith sample signal, a peak detection result representing a jth sampling point of an ith sample signal.

[0041] In a second aspect of the present application, a non-contact stress detection system based on a remote pulse wave signal is provided, and the system comprises:

[0042] a first acquisition module configured to acquire a remote pulse wave signal in a preset time period as a first signal;

[0043] a second acquisition module configured to acquire a first derivative of the first signal as a second signal;

[0044] a stress detection module configured to input the first signal and the second signal into a preset stress detection network, and acquire a binary classification result output by the stress detection network as a stress detection result;

[0045] wherein the stress detection network comprises a convolutional layer, a pooling layer, a peak attention module, a long short-term memory network, and a fully connected layer; the convolutional layer and the pooling layer are used for signal feature encoding, the peak attention module is used for introducing a peak detection auxiliary task and calculating time attention weights according to peak positions, the long short-term memory network is used for extracting time sequence information, and the fully connected layer is used for acquiring a stress state recognition result.

[0046] In a third aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0047] the memory is configured to store a computer program;

[0048] the processor is configured to execute the program stored on the memory, and implement any method step of the non-contact stress detection based on the remote pulse wave signal.

[0049] In a fourth aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which is executed by a processor to implement any method step of the non-contact stress detection based on the remote pulse wave signal.

[0050] (III) Beneficial Effects

[0051] The present application provides a non-contact stress detection method and system based on a remote pulse wave signal. Compared with the prior art, the following beneficial effects are achieved:

[0052] The method comprises: acquiring a remote pulse wave signal in a preset time period as a first signal; acquiring a first derivative of the first signal as a second signal;

[0053] The first signal and the second signal are input into a preset stress detection network, and a binary classification result output by the stress detection network is acquired as a stress detection result.

[0054] The stress detection network comprises a convolution layer, a pooling layer, a wave peak attention module, a long short-term memory network and a fully connected layer; the convolution layer and the pooling layer are used for signal feature coding, the wave peak attention module is used for introducing a wave peak detection auxiliary task and calculating time attention weights according to wave peak positions, the long short-term memory network is used for extracting time sequence information, and the fully connected layer is used for acquiring a stress state recognition result.

[0055] Based on the above processing, the technical scheme provided by the present application combines the rPPG signal and its first derivative signal as the basis for non-contact stress detection from two angles of physiological signal time sequence change characteristics and instantaneous change characteristics, automatically extracts features related to the stress state by using a CNN+LSTM network, different signal features form an advantage complementation, and compared with manually extracting heart rate variability features, the robustness to signal noise is stronger. In addition, the above processing process does not require the participation of professional technicians, can reduce the professional requirements for the operator, and improves the application range and detection efficiency of the non-contact stress detection. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0057] Figure 1 A flowchart of a non-contact stress detection method based on a remote pulse wave signal provided by an embodiment of the present application;

[0058] Figure 2 A network structure diagram of a stress detection network provided by an embodiment of the present application;

[0059] Figure 3 A structure diagram of a wave peak attention module provided by an embodiment of the present application;

[0060] Figure 4 A flowchart of another non-contact stress detection method based on a remote pulse wave signal provided by an embodiment of the present application;

[0061] Figure 5 A structural diagram of a non-contact stress detection system based on a remote pulse wave signal is provided for an embodiment of the present application.

[0062] Figure 6 A structural diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0063] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0064] The embodiments of the present application provide a non-contact stress detection method and system based on a remote pulse wave signal, solve the problems of skin contact discomfort and high professional requirement of operation existing in the existing stress state recognition method based on physiological signals, realize automatic extraction of features related to stress state by using a CNN+LSTM network, and form complementary advantages between different signal features. Compared with manual extraction of heart rate variability features, the robustness to signal noise is stronger. In addition, the above processing process does not require the participation of professional technicians, can reduce the professional requirement for the operator, and improves the application range of the non-contact stress detection scene.

[0065] The technical solutions in the embodiments of the present application are as follows to solve the above technical problems:

[0066] Most of the existing methods for recognizing stress state based on electrocardiogram signals and PPG signals are as follows: (1) signal peak value monitoring; (2) HRV feature statistics; and (3) using SVM or random forest method to predict stress state. However, the existing method architecture has the following deficiencies:

[0067] 1. Most of the existing stress state recognition methods relying on physiological signals use contact type devices such as electrocardiographs and bracelets. The contact type signal acquisition device is attached to the skin, which will cause discomfort and deformation of the blood vessels and other tissues at the contact part, affecting the monitoring result. In addition, the skin of some people may have allergic reactions to the basic materials of the contact type device.

[0068] 2. Existing non-contact stress assessment methods rely on detecting signal peaks from rPPG signals and further calculating the peak-recession interval (RR interval) to extract HRV features. HRV is a set of manually defined features for the RR interval, extracted through manual calculation. This feature extraction method requires extensive expertise and optimization in the stress detection field, making it difficult to capture the complexity and diversity of the data. Even experienced experts often struggle to ensure the extracted features are well-suited to the target task.

[0069] 3. Existing deep learning-based end-to-end stress assessment methods can extract necessary information on their own and learn complex and high-level representations of data features, but they cannot effectively combine prior knowledge in the field of stress detection. For example, the peak point of the pulse wave is a key signal point for stress state detection.

[0070] To address the aforementioned problems, this invention provides a non-contact stress detection method based on remote pulse wave signals. For example... Figure 1 As shown, the method includes the following steps:

[0071] S1. Acquire a remote pulse wave signal within a preset time period as the first signal.

[0072] S2. Obtain the first derivative of the first signal as the second signal.

[0073] S3. Input the first signal and the second signal into a preset stress detection network, and obtain the binary classification result output by the stress detection network as the stress detection result.

[0074] The stress detection network includes: convolutional layers, pooling layers, a peak attention module, a long short-term memory network, and a fully connected layer; the convolutional layers and pooling layers are used for signal feature encoding, the peak attention module is used to introduce peak detection auxiliary tasks and calculate time attention weights based on peak positions, the long short-term memory network is used to extract temporal information, and the fully connected layer is used to obtain stress state recognition results.

[0075] Based on the above processing, the technical solution provided by this invention addresses both the temporal and instantaneous variation characteristics of physiological signals. It combines the rPPG signal and its first derivative signal as the basis for non-contact stress detection, employing a CNN+LSTM network to automatically extract features related to the stress state. The different signal features complement each other, resulting in stronger robustness to signal noise compared to manually extracting heart rate variability features. Furthermore, the above processing does not require the involvement of professional technicians, reducing the professional requirements for operators and expanding the applicability of non-contact stress detection scenarios.

[0076] For step S1, the remote pulse wave signal, i.e., the rPPG signal in the present application. The rPPG signal is obtained as follows:

[0077] First, a visible light facial video of an observed user is captured by a visible light camera. The face detection and positioning in the visible light facial video is achieved by detecting the face part in the video and segmenting the face ROI region based on the FaceBoxes face detector. Then, the extracted face ROI region image is judged whether each pixel in the image is skin or not by using a YCbCr space skin adaptive segmentation method algorithm (where YCbCr is a coding of RGB color space and is widely used in digital video compression systems). If yes, the pixel value is retained, otherwise the pixel value is set to 0. The judgment rule is as follows:

[0078]

[0079] Wherein, pixel represents the pixel to be judged, skin represents that the pixel is judged as skin, non-skin represents that the pixel is judged as non-skin, Cr represents the red color component, Cb represents the blue color component, and y represents the brightness component.

[0080] After that, the image sequence after ROI segmentation and skin segmentation is used to extract the rPPG signal by using the POS (Plane-Orthogonal-to-Skin) algorithm. The POS extraction rPPG processing flow is as follows: for each image, the average value of each individual R, G and B channel is obtained, the average value is connected with the RGB channel average value of the previous image, and the 3-channel signal [R n ,G n ,B n ] is obtained by detrending and 3-order Butterworth band-pass filtering with a cutoff frequency of 0.7-2.5 Hz, and the obtained signal is divided by the average value to obtain a new signal

[0081] That is, the rPPG signal is obtained by the following steps:

[0082]

[0083]

[0084]

[0085] Wherein, S represents the extracted rPPG signal, X s and Y s are intermediate variables in the calculation process, std(·) is the standard deviation, represents the average value of the G channel, an average value of the B channel, an average value of the R channel. It is worth noting that the rPPG signal reflects the timing change of the state, and the first derivative of the signal reflects the instantaneous change rule of the state, so the rPPG signal and the first derivative of the rPPG signal are jointly used as input data of the algorithm of the stress state detection neural network in the present application. That is, in the technical scheme provided by the present application, the first signal is the rPPG signal, and the second signal is the first derivative of the rPPG signal.

[0086] The technical scheme provided by the present application uses the rPPG signal and the first derivative of the rPPG signal as the basis for non-contact stress state detection. The timing characteristics of the state change of the subject are extracted from the rPPG signal, and the timing characteristics of the instantaneous change of the state of the subject are extracted from the first derivative of the rPPG signal. The neural network is used, and the feature extraction is more convenient. The features extracted from the timing and instantaneous change are more comprehensive.

[0087] In some embodiments, the preset stress detection network processes the first signal and the second signal, including the following steps:

[0088] Step one, after inputting the first signal and the second signal, extracting the signal features of the first signal based on the preset first convolutional layer and the first pooling layer as the first feature map, and extracting the signal features of the second signal based on the preset second convolutional layer and the second pooling layer as the second feature map;

[0089] Step two, extracting the signal features of the first signal based on the preset third convolutional layer as the third feature map, and extracting the signal features of the third feature map based on the first wave peak attention module as the fourth feature map;

[0090] Step three, extracting the signal features of the second signal based on the preset fourth convolutional layer as the fifth feature map, and extracting the signal features of the fifth feature map based on the second wave peak attention module as the sixth feature map; wherein the fourth convolutional layer and the third convolutional layer have the same structure; the first wave peak attention module and the second wave peak attention module have the same structure and share parameters;

[0091] Step four, extracting features of the fourth feature map based on the preset fifth convolutional layer and the fifth pooling layer to obtain the seventh feature map, and performing feature fusion on the seventh feature map and the first feature map to obtain the eighth feature map;

[0092] Step five, extracting features of the sixth feature map based on the preset sixth convolutional layer and the sixth pooling layer to obtain the ninth feature map, and performing feature fusion on the ninth feature map and the second feature map to obtain the tenth feature map;

[0093] Step six, based on the preset seventh convolutional layer and the seventh pooling layer, the eighth feature map is extracted to obtain the eleventh feature map; and based on the preset eighth convolutional layer and the eighth pooling layer, the tenth feature map is extracted to obtain the twelfth feature map; wherein the structures of the first convolutional layer, the second convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer and the eighth convolutional layer are the same; the structures of the first pooling layer, the second pooling layer, the fifth pooling layer, the sixth pooling layer, the seventh pooling layer and the eighth pooling layer are the same;

[0094] Step nine, the eleventh feature map and the twelfth feature map are channel spliced, and the feature map after channel splicing is input into the classifier, so that the classifier outputs the two-class results of stress state recognition; wherein the classifier is composed of long short-term memory network and full connection network.

[0095] See Figure 2 , Figure 2 The network structure diagram of the stress detection network provided by the embodiment of the application, wherein, Figure 2 The different network layers in the stress detection network correspond to the first convolutional layer, the second pooling layer, the second convolutional layer, …, the eighth pooling layer and the eighth convolutional layer in the above network layers in the above steps in the order from left to right. Meanwhile, Figure 2 The processing process of the stress detection network for the first signal and the second signal in the stress detection network can be seen from the above steps one to nine.

[0096] In actual work process, the stress state detection neural network provided by the application is composed of convolutional layer (Convolutional Neural Networks, CNN), maximum pooling layer, peak attention module (Peak Detection and Attention Module, PDAM), long short-term memory network (Long Short Term Memory, LSTM), full connection layer and the like. Among them, the stress state detection neural network processes 60s long signal, i.e. rPPG signal and first derivative of rPPG signal, as two input signals of input end each time.

[0097] The stress state detection neural network provided by the application constructs two feature extraction networks with the same structure but different part weights. The feature extraction network is composed of a peak attention branch sharing part of the weight and an independent convolution branch.

[0098] The first layer convolution in the peak attention branch and the PDAM connected thereto are shared by two feature extraction network weights, and since the function of the peak attention branch is to detect the peaks of the signal and pay attention to them, the processing work and function are the same for the rPPG signal and the first derivative of the rPPG signal.

[0099] In addition, weight sharing can enable partial interaction between feature extraction networks processing different signals, and the information of different view signals is considered in the feature extraction process, so that the extracted feature information is more abundant.

[0100] Since the peak information of the rPPG signal and the first derivative of the rPPG signal is different, the output of the PDAM will be further encoded by an independent convolution layer and a pooling layer.

[0101] The independent convolution branch in the stress state detection neural network provided by the application is to retain the independence of feature extraction of different signals, and the feature extraction network extracts features from the rPPG signal and the first derivative of the rPPG signal.

[0102] In addition, in order to integrate the features of different extraction network branches for the same signal, the outputs of the two extraction network branches are added and then subjected to convolution and pooling to realize the extraction of the comprehensive features of the signal.

[0103] In the above stress state detection neural network, the features of the rPPG signal and the first derivative thereof (i.e., the first signal and the second signal in the application) are extracted by the feature extraction network, and the features of the two are spliced through channels and input into the classifier. The classifier is composed of an LSTM and a fully connected layer. The LSTM includes two hidden layers for extracting time sequence features, and the output H of the last hidden layer of the LSTM is input into the fully connected layer, and the output of the fully connected layer is obtained. The stress state determination result is a binary classification of whether it is a stress state.

[0104] Since the traditional method needs to first detect the peaks of the rPPG signal, then obtain the heart rate variability related indicators according to the peak interval, and then perform stress detection, the signal peaks are a strong basis for stress state detection. However, the previous manual feature engineering method not only consumes time and effort, but also leads to incomplete extracted features. Therefore, the application proposes a peak attention module, which functions as an auxiliary task of peak detection and combines the attention mechanism idea (peak position is given a higher weight to guide the network to pay attention), guides the network to learn the peak-related features, thereby avoiding the step of manually extracting features.

[0105] In some embodiments, the peak attention module provided by the present application comprises a peak detection branch, a max-pooling branch, a weight adjustment branch and an attention residual branch; wherein the peak attention module comprises a first peak attention module and a second peak attention module.

[0106] The processing process of the peak attention module on the input feature map comprises the following steps:

[0107] S201, obtaining a peak detection result corresponding to the feature map based on the peak detection branch; wherein the feature map comprises a third feature map and a fifth feature map;

[0108] S202, obtaining a global pooling weight coefficient of the feature map based on the max-pooling branch;

[0109] S203, obtaining a weight adjustment coefficient of the feature map based on the weight adjustment branch;

[0110] S204, performing feature fusion on the peak detection result, the global pooling weight coefficient and the weight adjustment coefficient to obtain a comprehensive weight coefficient;

[0111] S205, performing mapping processing on the comprehensive weight coefficient based on a SoftMax function to obtain a time attention weight;

[0112] S206, multiplying the attention weight with the feature map in the time domain to obtain a feature map with enhanced time importance;

[0113] S207, performing feature fusion on the feature map with enhanced time importance and the feature map to obtain a fourth feature map or a sixth feature map.

[0114] Referring to Figure 3 , Figure 3 A structure diagram of a peak attention module provided by an embodiment of the present application. As shown in Figure 3 , the peak attention module can be divided into a peak detection branch, a max-pooling branch, a weight adjustment branch and an attention residual branch according to the order from top to bottom. Among them, the peak detection branch itself has two branches, each branch has a different receptive field, which can encode features in the signal time domain at different scales, realize higher precision peak detection, and realize time sequence feature fusion of different scales by adding the two branches and passing through a layer of convolution. The output is normalized to 0 to 1 by a Sigmoid function to obtain a peak detection result, which will be one of the bases for calculating the attention weight. In addition, the peak position detection task can also assist in stress state detection.

[0115] In some embodiments, step S201 comprises the following contents:

[0116] S20101, large-scale receptive field feature extraction is performed on the feature map based on a double-layer one-dimensional convolution layer and a LeakyReLU function to obtain first to-be-processed wave peak features, and small-scale receptive field feature extraction is performed on the feature map based on a single-layer one-dimensional convolution layer and a LeakyReLU function to obtain second to-be-processed wave peak features. The large-scale receptive field means that the receptive field is larger than the feature region to be extracted, and the small-scale receptive field means that the receptive field is smaller than the feature region to be extracted.

[0117] S20102, the first to-be-processed wave peak feature and the second to-be-processed wave peak feature are added to obtain third to-be-processed wave peak features containing multi-scale receptive fields.

[0118] S20103, wave peak detection calculation is performed on the third to-be-processed wave peak features based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal as a wave peak detection result.

[0119] The maximum pooling branch selects a point with a global maximum value for different channels of the input feature map of the module, realizes feature compression, and retains key features of the original input feature map. The branch also normalizes the output to 0 to 1 through a Sigmoid function, and the result is also one of the calculation bases of the attention weight.

[0120] Based on the above processing, the PDAM proposed in the application combines the multi-task learning and attention mechanism ideas, combines the domain prior knowledge, guides the neural network to learn the rPPG signal and the first derivative wave peak feature, and effectively improves the recognition accuracy.

[0121] In some embodiments, step S202 includes the following content:

[0122] S20201, global pooling weight calculation is performed on the feature map based on a global maximum pooling layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal, that is, a global pooling weight coefficient.

[0123] The weight adjustment branch gives the wave peak attention module the freedom to focus on other key points in the signal, that is, in addition to the peak point, the network can pay attention to other key points that are beneficial to stress detection. The branch is also processed through a Sigmoid function, and is one of the calculation bases of the attention weight.

[0124] In some embodiments, step S203 includes the following content:

[0125] S20301, adjustment weight calculation is performed on the feature map based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal, that is, a weight adjustment coefficient.

[0126] It is worth noting that the outputs of the above three branches are added and then passed through a SoftMax function to calculate the attention weight, the attention residual branch multiplies the input feature map of the module with the calculated weight in the time domain to realize feature attention to the key points in the time domain, and adds the multiplied result to the original feature map to retain the original feature information.

[0127] Based on the above processing, the rPPG signal and its first derivative signal wave peak are taken as key prior knowledge (the traditional method needs to perform wave peak detection first, then obtain the heart rate variability related index according to the wave peak interval, and then perform stress detection), combined with the multi-task learning and attention mechanism idea, and the neural network is guided to learn the prior knowledge.

[0128] In Figure 3 , R peak is the wave peak detection result, which is a continuous sequence with values in the range of 0-1 and the same length as the input signal, and the points closer to 1 are more likely to be wave peak points; R output is the output result of the wave peak attention module; let the input feature map of the wave peak attention module be f rPPG , then

[0129]

[0130] wherein Ol represents a max-pooling layer, Conv represents a one-dimensional convolution layer, sigma(·) represents a Sigmoid function, represents element-level multiplication, represents element-level addition.

[0131] In the actual working process, the non-contact stress detection process based on the remote pulse wave signal provided by the application is as shown in Figure 4 . At the same time, the technical scheme provided by the application can realize stress state detection only by using an ordinary camera, which is convenient and fast to deploy and has little interference to users. Only physiological signals extracted from facial images are used, compared with methods using behavioral features such as expressions and actions (which can be actively controlled), the data used is more objective, which helps to improve the accuracy of non-contact stress detection.

[0132] In some embodiments, the loss function of the stress detection network is:

[0133] wherein Loss stress represents the loss value of stress state determination, N represents the total number of samples, i represents the i-th sample, represents the binary classification result of the stress detection network corresponding to the sample i, represents the stress state label corresponding to the sample i.

[0134] a peak detection loss value representing the first signal, a peak detection loss value representing the second signal, and wherein Lpss peak comprises or L represents the total number of signal sampling points of a single sample, and j represents the jth sampling point of the signal; a peak label representing the jth sampling point of the ith sample signal, a peak detection result representing the jth sampling point of the ith sample signal.

[0135] In actual work, Loss stress For stress state determination loss, cross-entropy is used as the loss function; and are the peak detection losses of the rPPG signal and the first derivative signal, respectively, since whether a signal sampling point belongs to a peak can also be regarded as a binary classification problem, the peak detection loss function also uses cross-entropy. In addition, T stress and T peak are the stress state label and the signal peak label, respectively. The signal peak label is obtained by the find_peaks method provided by the scipy.signal toolkit.

[0136] In order to verify the effectiveness of the non-contact stress detection method provided by the application, the test data set UBFC-Phys (public data set) is used, and the technical solutions of the application, the technical solutions provided by the application (remove the peak attention module), the technical solutions provided by the application (remove the first derivative branch), and the data set baseline manual feature extraction SVM method (wherein the kernel function is rbf) in the prior art are compared by using the 10-fold cross-validation method. The results are shown in Table 1.

[0137] Table 1 Experimental results table

[0138]

[0139]

[0140] According to the content of Table 1, the accuracy, recall rate, precision and F1 value of the non-contact stress detection method provided by the application after experimental detection are all higher than those of other methods, thereby effectively verifying the effectiveness of the non-contact stress detection method provided by the application.

[0141] Referring to Figure 5The present invention also provides a non-contact stress detection system based on remote pulse wave signals, the system comprising:

[0142] The first acquisition module 501 is used to acquire a remote pulse wave signal over a preset time period as a first signal;

[0143] The second acquisition module 502 is used to acquire the first derivative of the first signal as the second signal;

[0144] The stress detection module 503 is used to input the first signal and the second signal into a preset stress detection network, and obtain the binary classification result output by the stress detection network as the stress detection result.

[0145] The stress detection network includes: convolutional layers, pooling layers, a peak attention module, a long short-term memory network, and a fully connected layer; the convolutional layers and pooling layers are used for signal feature encoding, the peak attention module is used to introduce peak detection auxiliary tasks and calculate time attention weights based on peak positions, the long short-term memory network is used to extract temporal information, and the fully connected layer is used to obtain stress state recognition results.

[0146] It is understood that the non-contact stress detection system based on remote pulse wave signals provided in this embodiment of the invention corresponds to the non-contact stress detection method based on remote pulse wave signals described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the non-contact stress detection method based on remote pulse wave signals, and will not be repeated here.

[0147] This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0148] Memory 603 is used to store computer programs;

[0149] The processor 601, when executing the program stored in the memory 603, implements any of the above-mentioned non-contact stress detection methods based on remote pulse wave signals.

[0150] It is to be noted that, in the present text, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0151] The above examples are merely used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A non-contact stress detection method based on a remote pulse wave signal, characterized by, The method comprises: acquiring a remote pulse wave signal in a preset time period as a first signal; acquiring a first derivative of the first signal as a second signal; inputting the first signal and the second signal into a preset stress detection network, and acquiring a binary classification result output by the stress detection network as a stress detection result; wherein the stress detection network comprises a convolutional layer, a pooling layer, a wave peak attention module, a long short-term memory network, and a fully connected layer; the convolutional layer and the pooling layer are used for signal feature coding, the wave peak attention module is used for introducing a wave peak detection auxiliary task and calculating time attention weights according to wave peak positions, the long short-term memory network is used for extracting time sequence information, and the fully connected layer is used for acquiring a stress state recognition result.

2. The method of claim 1, wherein, The processing process of the preset stress detection network on the first signal and the second signal comprises: after inputting the first signal and the second signal, extracting signal features of the first signal as a first feature map based on a preset first convolutional layer and a first pooling layer, and extracting signal features of the second signal as a second feature map based on a preset second convolutional layer and a second pooling layer; extracting signal features of the first signal as a third feature map based on a preset third convolutional layer, and extracting signal features of the third feature map as a fourth feature map based on a first wave peak attention module; extracting signal features of the second signal as a fifth feature map based on a preset fourth convolutional layer, and extracting signal features of the fifth feature map as a sixth feature map based on a second wave peak attention module; wherein the fourth convolutional layer has the same structure as the third convolutional layer; the first wave peak attention module has the same structure as the second wave peak attention module and shares parameters with the second wave peak attention module; extracting features of the fourth feature map based on a preset fifth convolutional layer and a fifth pooling layer to obtain a seventh feature map, and fusing features of the seventh feature map and the first feature map to obtain an eighth feature map; extracting features of the sixth feature map based on a preset sixth convolutional layer and a sixth pooling layer to obtain a ninth feature map, and fusing features of the ninth feature map and the second feature map to obtain a tenth feature map; extracting features of the eighth feature map based on a preset seventh convolutional layer and a seventh pooling layer to obtain an eleventh feature map, and extracting features of the tenth feature map based on a preset eighth convolutional layer and an eighth pooling layer to obtain a twelfth feature map; wherein the first convolutional layer, the second convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, and the eighth convolutional layer all have the same structure; the first pooling layer, the second pooling layer, the fifth pooling layer, the sixth pooling layer, the seventh pooling layer, and the eighth pooling layer all have the same structure; channel splicing the eleventh feature map and the twelfth feature map, and inputting the feature map after channel splicing into a classifier to make the classifier output a binary classification result of stress state recognition; wherein the classifier is composed of a long short-term memory network and a fully connected network.

3. The method of claim 2, wherein, The wave peak attention module comprises a peak detection branch, a max pooling branch, a weight adjustment branch and an attention residual branch; wherein, the wave peak attention module comprises a first wave peak attention module and a second wave peak attention module; The processing process of the wave peak attention module on the input feature map comprises: obtaining a wave peak detection result corresponding to the feature map based on the peak detection branch; wherein, the feature map comprises a third feature map and a fifth feature map; obtaining a global pooling weight coefficient of the feature map based on the max pooling branch; obtaining a weight adjustment coefficient of the feature map based on the weight adjustment branch; performing feature fusion on the wave peak detection result, the global pooling weight coefficient and the weight adjustment coefficient to obtain a comprehensive weight coefficient; performing mapping processing on the comprehensive weight coefficient based on a SoftMax function to obtain a time attention weight; multiplying the attention weight with the feature map in the time domain to obtain a feature map with enhanced time importance; performing feature fusion on the feature map with enhanced time importance and the feature map to obtain a fourth feature map or a sixth feature map.

4. The method of claim 3, wherein, The process of obtaining a wave peak detection result corresponding to the feature map based on the peak detection branch comprises: performing large-scale receptive field feature extraction on the feature map based on a double-layer one-dimensional convolution layer and a LeakyReLU function to obtain a first to-be-processed wave peak feature, and performing small-scale receptive field feature extraction on the feature map based on a single-layer one-dimensional convolution layer and a LeakyReLU function to obtain a second to-be-processed wave peak feature; performing addition processing on the first to-be-processed wave peak feature and the second to-be-processed wave peak feature to obtain a third to-be-processed wave peak feature containing multi-scale receptive fields; performing wave peak detection calculation on the third to-be-processed wave peak feature based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal as the wave peak detection result.

5. The method of claim 3, wherein, The process of obtaining a global pooling weight coefficient of the feature map based on the max pooling branch comprises: performing global pooling weight calculation on the feature map based on a global max pooling layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal, i.e. the global pooling weight coefficient.

6. The method of claim 3, wherein, The process of obtaining a weight adjustment coefficient of the feature map based on the weight adjustment branch comprises: performing adjustment weight calculation on the feature map based on a one-dimensional convolution layer and a Sigmoid function to obtain a one-dimensional array with the same length as the input signal, i.e. the weight adjustment coefficient.

7. The method of claim 1, wherein, The loss function of the preset stress detection network is: wherein, Loss stress represents a loss value of stress state determination, N represents the total number of samples, i represents the ith sample, represents the binary classification result of the stress detection network corresponding to the sample i, represents the stress state label corresponding to the sample i; a peak detection loss value of the first signal, a peak detection loss value of the second signal, and wherein Loss peak comprises or L represents the total number of signal sampling points of a single sample, and j represents the jth sampling point of the signal; a peak label of the jth sampling point of the ith sample signal, a peak detection result of the jth sampling point of the ith sample signal.

8. A non-contact stress detection system based on a remote pulse wave signal, characterized by, The system comprises: a first obtaining module configured to obtain a remote pulse wave signal in a preset time period as a first signal; a second obtaining module configured to obtain a first derivative of the first signal as a second signal; a stress detection module configured to input the first signal and the second signal into a preset stress detection network, and obtain a binary classification result output by the stress detection network as a stress detection result; The stress detection network comprises a convolutional layer, a pooling layer, a wave peak attention module, a long short-term memory network and a full connection layer; the convolutional layer and the pooling layer are used for signal feature coding, the wave peak attention module is used for introducing a wave peak detection auxiliary task and calculating time attention weights according to wave peak positions, the long short-term memory network is used for extracting time sequence information, and the full connection layer is used for obtaining a stress state recognition result.

9. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory is used for storing a computer program. The processor is used for executing the program stored on the memory to realize the method steps in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-7.

Images