Physiological signal processing method, device and medium

Abstract

The present application discloses a physiological signal processing method, device and medium, which are applied to the field of physiological signal processing technology. The method includes: obtaining long-sequence video data of the face; processing the long-sequence video data of the face to obtain the spatiotemporal characteristics of the long-sequence video data of the face; obtaining initial fast features and initial slow features according to the spatiotemporal characteristics, wherein the initial fast features and initial slow features are respectively features representing different time dimensions; according to the initial fast features, combined with a time difference module, a deep fast feature is obtained, and the time difference module is used for feature extraction based on a preset state space model; according to the initial fast features and the initial slow features, combined with a time difference module, a deep slow feature is obtained; according to the deep fast features and the deep slow features, a physiological signal is obtained. The present application can effectively improve the accuracy and reliability of physiological signals.

Description

Technical Field

[0001] The present application relates to the technical field of physiological signal processing, in particular to a physiological signal processing method, device and medium. Background Art

[0002] Remote Photo-PlethysmoGraphy (rPPG) is a non-contact physiological signal measurement technology that uses sensors such as cameras to capture the periodic changes in skin color caused by the cardiac cycle, and then extracts blood volume pulse signals and measures physiological indicators related to the cardiac cycle such as heart rate, respiratory rate and heartbeat variability. At present, facial video analysis has become the main way to extract rPPG signals. rPPG signals refer to physiological signals based on remote photoplethysmography. Related technologies are usually based on deep learning technologies such as convolutional neural networks and Transformer to achieve the reconstruction of rPPG signals, and the extraction accuracy of rPPG signals needs to be improved. Summary of the invention

[0003] The embodiments of the present application provide a physiological signal processing method, device and medium for improving the reliability and accuracy of physiological signals.

[0004] On the one hand, the present invention provides a method for processing physiological signals, comprising the following steps:

[0005] Obtain facial long sequence video data;

[0006] Processing the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data;

[0007] According to the spatiotemporal features, an initial fast feature and an initial slow feature are obtained; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively;

[0008] According to the initial fast features, combined with the time series difference module, a deep fast feature is obtained; wherein the time series difference module is used to extract features based on a preset state space model;

[0009] According to the initial fast feature and the initial slow feature, combined with the time series difference module, a deep slow feature is obtained;

[0010] A physiological signal is obtained according to the deep fast feature and the deep slow feature.

[0011] On the other hand, an embodiment of the present application provides a physiological signal processing device, comprising:

[0012] An acquisition module is used to acquire facial long sequence video data;

[0013] A first processing module is used to process the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data;

[0014] A second processing module is used to obtain an initial fast feature and an initial slow feature according to the spatiotemporal feature; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively;

[0015] A third processing module is used to obtain deep fast features based on the initial fast features in combination with a time series difference module; wherein the time series difference module is used to extract features based on a preset state space model;

[0016] A fourth processing module, configured to obtain a deep slow feature according to the initial fast feature and the initial slow feature in combination with the time series difference module;

[0017] The fifth processing module is used to obtain a physiological signal according to the deep fast feature and the deep slow feature.

[0018] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned physiological signal processing method.

[0019] The beneficial effects of the present application are: providing a physiological signal processing method, device and medium, firstly, obtaining long-sequence video data of the face; secondly, processing the long-sequence video data of the face to obtain the spatiotemporal features of the long-sequence video data of the face; then, according to the spatiotemporal features, obtaining initial fast features and initial slow features, which can effectively and accurately capture the slowly changing physiological features and the rapidly changing physiological features in the spatiotemporal features, the slowly changing physiological features are the initial slow features, the rapidly changing physiological features are the rapidly changing physiological features, the initial fast features and the initial slow features represent different time dimensions respectively; then, according to the initial fast features, combined with the time difference module, obtaining The deep fast features and the temporal difference module are used for feature extraction based on a preset state-space model, which can effectively capture the long-distance spatiotemporal dependencies in the rapidly changing physiological features, and the deep slow features are obtained in combination with the temporal difference module based on the initial fast features and the initial slow features. This can not only effectively capture the long-distance spatiotemporal dependencies in the slowly changing physiological features, but also realize the effective fusion of physiological features in different time dimensions, reduce feature redundancy, and ensure feature accuracy. Finally, the physiological signals are obtained based on the deep fast features and the deep slow features, and the extraction and detection of rPPG signals are realized, thereby effectively improving the accuracy and reliability of the physiological signals.

[0020] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the physiological signal processing method provided by the present application;

[0022] Figure 2 is a schematic diagram of the physiological signal processing method provided by the present application;

[0023] Figure 3 It is a structural diagram of the physiological signal processing device provided in this application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0028] Remote photoplethysmography is a non-contact physiological signal measurement technology that uses sensors such as cameras to capture the periodic changes in skin color caused by the cardiac cycle, and then extracts the blood volume pulse signal (Blood Volume Pulse, BVP), and measures physiological indicators related to the cardiac cycle such as heart rate, respiratory rate and heartbeat variability. Unlike traditional electrocardiograms (ECG) and photoplethysmography (PPG), remote photoplethysmography can use ordinary cameras to detect changes in light absorption and reflection caused by blood flow, thus providing a more convenient and comfortable way to monitor physiological signals.

[0029] Currently, facial video analysis has become the main method for extracting rPPG signals, which refers to physiological signals based on remote photoplethysmography.

[0030] In the early development stage of remote photoplethysmography, some early technologies used traditional signal processing technology to track color changes in a specific region of interest (ROI) on the face and obtain rPPG signals based on the color changes. Although these early technologies have achieved certain success, they face many challenges in practical applications. For example, they are affected by other external factors such as ambient light fluctuations and facial movements, and will introduce inevitable noise during signal processing, which will affect the accuracy of rPPG signals.

[0031] With the rise of deep learning technology, related technologies are based on deep learning technologies such as Convolutional Neural Network (CNN) and Transformer to extract spatial and temporal features in facial videos, thereby reconstructing rPPG signals. Related technologies can improve the accuracy of rPPG signals to a certain extent.

[0032] However, first of all, rPPG signals often involve complex spatiotemporal changes, and it is crucial to capture long-range spatiotemporal dependencies in facial videos. Long-range spatiotemporal dependencies can be understood as the relationship between long-range temporal features and spatial features; while convolutional neural networks usually only rely on the extraction of local spatial features in facial videos, and their performance in capturing long-range spatiotemporal dependencies is insufficient, which in turn leads to insufficient accuracy of rPPG signals.

[0033] Secondly, although the self-attention mechanism in Transformer can effectively capture the global features in facial videos, the computational complexity of Transformer increases significantly when processing long-sequence facial video data, which will make it difficult for Transformer to efficiently focus on local information related to rPPG signals in facial videos, thereby affecting the accuracy of rPPG signals.

[0034] In view of this, the embodiments of the present application provide a physiological signal processing method, device and medium, aiming to improve the accuracy of rPPG signals.

[0035] First, the specific implementation steps of the physiological signal processing method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0036] The physiological signal processing method provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0037] Reference Figure 1 , Figure 1 : is a flow chart of the physiological signal processing method provided by the present application. The physiological signal processing method may include the following steps S101-S106:

[0038] S101, obtaining long sequence video data of the face;

[0039] S102, processing the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data;

[0040] S103, obtaining an initial fast feature and an initial slow feature according to the spatiotemporal feature; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively;

[0041] S104, obtaining deep fast features based on the initial fast features and in combination with a temporal difference module; wherein the temporal difference module is used to extract features based on a preset state space model;

[0042] S105, obtaining a deep slow feature based on the initial fast feature and the initial slow feature in combination with a time series difference module;

[0043] S106, obtaining a physiological signal according to the deep fast feature and the deep slow feature.

[0044] In the embodiment of the present application, first, a long sequence video data of the face is obtained; secondly, the long sequence video data of the face is processed to obtain the spatiotemporal features of the long sequence video data of the face, so as to realize preliminary feature extraction; then, according to the spatiotemporal features, initial fast features and initial slow features are obtained, which can effectively and accurately capture the slowly changing physiological features and the rapidly changing physiological features in the spatiotemporal features, the slowly changing physiological features are the initial slow features, the rapidly changing physiological features are the rapidly changing physiological features, and the initial fast features and the initial slow features represent different time dimensions respectively; then, according to the initial fast features, a deep fast feature is obtained in combination with a time difference module, and the time difference module is used to obtain a deep fast feature based on a preset state space model (State Space The feature extraction of the deep fast feature and the deep slow feature is carried out based on the initial fast feature and the initial slow feature, which can effectively capture the long-distance spatiotemporal dependency in the physiological features with slow changes, and the deep slow feature is obtained by combining the time difference module. It can not only effectively capture the long-distance spatiotemporal dependency in the physiological features with slow changes, but also realize the effective fusion of physiological features of different time dimensions, reduce feature redundancy, and ensure feature accuracy; finally, the physiological signal is obtained based on the deep fast feature and the deep slow feature, and then the extraction and detection of the rPPG signal is realized. In this way, the embodiment of the present application can effectively improve the accuracy and reliability of the physiological signal.

[0045] In the above step S101, long sequence video data of the face to be tested is obtained to realize the detection of rPPG signals.

[0046] The above-mentioned long facial sequence video data may include several frames of facial images, and the facial images show the face of the user.

[0047] For example, the above-mentioned long facial sequence video data may include 128 frames of facial images, the frame height and frame width of each facial image frame are both 128, and each facial image frame contains three primary color channels of red, green and blue.

[0048] The above-mentioned facial long sequence video data is used to represent facial video data whose video duration is greater than a preset duration threshold.

[0049] The above-mentioned duration threshold can be set according to actual conditions, and the embodiments of the present application do not make any specific limitations on this.

[0050] The above-mentioned acquisition of the long-sequence video data of the face may include photographing the user's face through an image acquisition module such as a camera to obtain the long-sequence video data of the face, but is not limited thereto.

[0051] In the above step S102, the long facial sequence video data is subjected to processes such as differential normalization and preliminary feature extraction to obtain the spatiotemporal features of the long facial sequence video data.

[0052] The above-mentioned spatiotemporal features may include temporal features and spatial features in long-sequence facial video data.

[0053] The above-mentioned processing of the long facial sequence video data to obtain the spatiotemporal characteristics of the long facial sequence video data may include using a preset first neural network model to perform feature extraction on the long facial sequence video data to obtain the temporal characteristics of the long facial sequence video data, and using a preset second neural network model to perform feature extraction on the long facial sequence video data to obtain the spatial characteristics of the long facial sequence video data, but is not limited to this.

[0054] The above-mentioned first neural network model and the above-mentioned second neural network model can be set according to actual conditions, and the embodiments of the present application do not make specific limitations on this.

[0055] For example, the first neural network model may be a long short-term memory neural network (LSTM) model, and the second neural network model may be a convolutional neural network model, but is not limited thereto.

[0056] In the above step S103, the spatiotemporal features of the long face sequence video data are processed to obtain features representing different time dimensions, namely, initial fast features and initial slow features.

[0057] The above-mentioned obtaining of the initial fast feature and the initial slow feature based on the spatiotemporal features may include obtaining preset feature mapping data, searching out the fast feature corresponding to the spatiotemporal feature from the preset feature mapping data as the initial fast feature, and searching out the slow feature corresponding to the spatiotemporal feature from the preset feature mapping data as the initial slow feature, wherein the feature mapping data includes several preset spatiotemporal features and the fast feature and the slow feature corresponding to each spatiotemporal feature, but is not limited to this.

[0058] The above initial rapid features are used to indicate the physiological features that change rapidly in spatiotemporal characteristics.

[0059] The above initial slow features are used to indicate physiological features of slow changes in spatiotemporal features.

[0060] In the above step S104, the initial fast features are processed by the temporal difference module to capture the long-distance spatiotemporal dependencies in the rapidly changing physiological features, thereby obtaining deep fast features.

[0061] The above-mentioned temporal difference module may be a module having at least one state space model, which is used for performing feature extraction based on a preset state space model.

[0062] The above-mentioned state space model can be set according to actual conditions, and the embodiments of the present application do not make any specific limitations on this.

[0063] For example, the state space model may be a linear Gaussian state space model (Linear Gaussian State Space Model, LGSSM); or, the state space model may be a Mamba model, but is not limited thereto.

[0064] The above-mentioned method of obtaining deep fast features based on the initial fast features in combination with the time difference module may include inputting the initial fast features into the time difference module, processing the initial fast features through the linear Gaussian state space model in the time difference module, and obtaining deep fast features, but is not limited to this.

[0065] In the above step S105, the initial slow features are processed by the temporal difference module to capture the long-distance spatiotemporal dependencies in the slowly changing physiological features, and the physiological features of different time dimensions are fused to obtain deep slow features.

[0066] The above-mentioned method of obtaining deep slow features based on the initial fast features and the initial slow features in combination with the time difference module may include fusing the initial slow features and the initial fast features to obtain initial fused features as input of the time difference module, and processing the initial fused features through the linear Gaussian state space model in the time difference module to obtain deep slow features, but is not limited to this.

[0067] The above step S104 and the above step S105 may be performed in parallel, for example, the above step S104 and the above step S105 are performed at the same time.

[0068] Alternatively, the step S104 and the step S105 may be performed in series, for example, the step S104 is performed first, and then the step S105 is performed, or for example, the step S105 is performed first, and then the step S104 is performed.

[0069] In the above step S106, after obtaining the deep fast features and the deep slow features, signal extraction is performed based on the deep fast features and the deep slow features to obtain the physiological signal.

[0070] The above-mentioned physiological signals may include rPPG signals.

[0071] The above-mentioned obtaining of physiological signals based on deep fast features and deep slow features may include fusing the deep fast features and the deep slow features to obtain fused deep features, and performing prediction processing on the fused deep features through a preset predictor to obtain the physiological signals, but is not limited to this.

[0072] The following will be combined Figure 2 The above steps are further explained.

[0073] In some implementations, the above-mentioned acquisition of facial long sequence video data may include:

[0074] Acquire face video data, where the face video data includes a plurality of frames of initial face images;

[0075] Performing face detection on the first frame initial face image in the face video data to obtain the face area in the first frame initial face image;

[0076] Each frame of the initial face image is cropped based on the facial region to obtain each frame of the facial image;

[0077] Through each frame of facial image, a long facial sequence video data is constructed.

[0078] In this embodiment, first, face video data including several frames of initial face images are obtained, and the initial face images show the face of the user; then, face detection is performed on the first frame of the initial face image in the face video data to obtain the face area in the first frame of the initial face image, and the face area refers to the rectangular frame area covering the face, and the face area is used to implement the cropping process; then, for each frame of the initial face image, the initial face image is cropped using the face area, aiming to crop the user's face from the initial face image, and then obtain the face image; by traversing each frame of the initial face image, each frame of the face image can be obtained; finally, through each frame of the face image, a long sequence video data of the face is constructed. In this way, it is beneficial to reduce the background noise in the video data and improve the quality of the video data.

[0079] The above-mentioned acquisition of facial video data may include photographing the user's face through an image acquisition module such as a camera to obtain the facial video data, but is not limited thereto.

[0080] Alternatively, the above-mentioned acquisition of facial video data may include acquiring facial video data through big data technologies such as web crawlers, but is not limited to this.

[0081] The above-mentioned facial detection on the first frame initial facial image in the facial video data to obtain the facial area in the first frame initial facial image may include using a preset third neural network model to perform facial detection on the first frame initial facial image in the facial video data to obtain the facial area in the first frame initial facial image, but is not limited to this.

[0082] The third neural network model mentioned above can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0083] The third neural network model may be a multi-task convolutional neural network (MTCNN) model; or, the third neural network model may be FaceNet, but is not limited thereto.

[0084] The above-mentioned cropping of each frame of initial face image based on the facial area to obtain each frame of facial image can include obtaining the coordinates of the facial area in the first frame of initial face image as the target coordinates; for each frame of initial face image, cropping the area whose coordinate information is the target coordinates in the initial face image as the facial image, but is not limited to this.

[0085] In some implementations, the processing of the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data may include:

[0086] Performing differential normalization processing on the long facial sequence video data to obtain differential video frames of the long facial sequence video data;

[0087] The differential video frame is subjected to at least one convolution process to obtain the spatiotemporal features of the facial long sequence video data.

[0088] In this embodiment, after obtaining the long-sequence video data of the face, first, the video data is subjected to differential normalization processing to reduce the influence of background pixels on the detection or extraction of rPPG signals, thereby obtaining differential video frames of the long-sequence video data of the face; then, the differential video frames are subjected to at least one convolution processing to obtain the spatiotemporal features of the long-sequence video data of the face, so as to realize preliminary feature extraction.

[0089] The above-mentioned differential normalization processing of the face long sequence video data may include performing differential processing on the face long sequence video data to obtain the face long sequence video data after differential processing, and then normalizing the face long sequence video data after differential processing to obtain differential video frames of the face long sequence video data, but is not limited to this.

[0090] For example, the above-mentioned differential normalization processing of the face long-sequence video data includes, for two adjacent frames in the face long-sequence video data, calculating the difference between the current frame and the previous frame as the first difference, calculating the difference between the previous frame and the current frame as the second difference, and calculating the ratio of the first difference to the second difference as the difference value between two adjacent frames; obtaining the difference values ​​of several adjacent frames by traversing all frames of the face long-sequence video data; determining the standard deviation of the difference values ​​of the face long-sequence video data based on the difference values ​​of several adjacent frames, and normalizing the standard deviation, thereby obtaining the differential video frames of the face long-sequence video data.

[0091] The above-mentioned convolution processing of the differential video frames at least once to obtain the spatiotemporal characteristics of the long facial sequence video data may include performing convolution processing on the differential video frames once through a preset fourth neural network model to obtain the spatiotemporal characteristics of the long facial sequence video data, but is not limited to this.

[0092] The fourth neural network model mentioned above can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0093] For example, the fourth neural network model mentioned above can be a convolutional neural network model.

[0094] In one example, the differential video frame of the above-mentioned facial long sequence video data can be expressed as , is the length of the differential video frame, is the frame height of the differential video frame, is the frame width of the differential video frame, is the number of three primary color channels.

[0095] The length, frame height, frame width and number of primary color channels of the differential video frame can be set according to actual conditions, and this embodiment does not specifically limit this. For example , , , , but not limited to this.

[0096] In some implementations, performing at least one convolution process on the differential video frame to obtain the spatiotemporal features of the facial long sequence video data may include:

[0097] The difference video frames are convolved through a shallow backbone network to obtain the spatiotemporal features of the facial long sequence video data;

[0098] Among them, the shallow backbone network includes three sequentially connected convolution modules. The first convolution module and the last convolution module both include sequentially connected convolution blocks, pooling layers, normalization layers and activation function layers. The second convolution module includes sequentially connected convolution blocks, normalization layers and activation function layers. The convolution kernel sizes of each convolution block are different. Among them, the convolution block is used to convolve the input data of the convolution module, the pooling layer is used to pool the output data of the convolution block, the normalization layer is used to normalize the output data of the pooling layer, and the activation function layer is used to delinearize the output data of the normalization layer to obtain the output data of the convolution module.

[0099] In this embodiment, a shallow backbone network is provided to realize preliminary feature extraction. Specifically, the shallow backbone network may include three sequentially connected convolution modules, the input of the first convolution module is a differential video frame, the output of the first convolution module is used as the input of the second convolution module, the output of the second convolution module is used as the input of the last convolution module, the output of the last convolution module is the spatiotemporal features of the facial long sequence video data, the first convolution module and the last convolution module both include sequentially connected convolution blocks, pooling layers, normalization layers, and activation function layers, the second convolution module includes sequentially connected convolution blocks, normalization layers, and activation function layers, and the convolution kernel sizes of each convolution block are different.

[0100] In the shallow backbone network, each convolution block is used to perform convolution processing on the input data of the convolution module based on the convolution kernel size of each convolution block to achieve feature extraction; each pooling layer is used to perform pooling processing on the output data of the convolution block to reduce the spatial dimension of the output data; each normalization layer is used to normalize the output data of the pooling layer to improve the generalization ability of the shallow backbone network; each activation function layer is used to perform nonlinear processing on the output data of the normalization layer based on a preset activation function to obtain the output data of the convolution module to enhance the feature expression ability of the shallow backbone network.

[0101] The convolution kernel size of the above convolution block can be set according to actual conditions, and this embodiment does not specifically limit this.

[0102] For example, the convolution kernel size of the convolution block of the first convolution module above is 1 5 5; The convolution kernel size of the convolution block of the second convolution module is 3 3 3; The convolution kernel size of the convolution block of the third convolution module is 3 3 3, but not limited to this.

[0103] The normalization layer of the above-mentioned convolution module may be a batch normalization layer; or, the normalization layer of the above-mentioned convolution module may be a layer normalization layer, but is not limited thereto.

[0104] The pooling layer of the above-mentioned convolution module may be a maximum pooling layer; or, the pooling layer of the above-mentioned convolution module may be an average pooling layer, but is not limited thereto.

[0105] The activation function in the activation function layer of the above-mentioned convolution module can be set according to actual conditions, and this embodiment does not make any specific limitation on this.

[0106] For example, the activation function in the above activation function layer may be a ReLU activation function; or, the activation function in the above activation function layer may be a Leaky ReLU activation function, but is not limited thereto.

[0107] In one example, the above spatiotemporal features can be expressed as , is the number of channels of spatiotemporal features, is the length of the space-time feature, is the height of the spatiotemporal feature, and , is the width of the spatiotemporal feature, and .

[0108] The number of channels of the above-mentioned spatiotemporal features can be set according to actual conditions, and this embodiment does not specifically limit this. For example , but not limited to this.

[0109] In some implementations, obtaining the initial fast feature and the initial slow feature according to the spatiotemporal feature may include:

[0110] Based on the preset downsampling ratio, the spatiotemporal features are subjected to time series downsampling processing to obtain initial fast features and initial slow features.

[0111] In this embodiment, the spatiotemporal features are sequentially downsampled using a preset downsampling ratio, aiming to capture the slowly changing physiological features and the rapidly changing physiological features in the spatiotemporal features from the time dimension, and then obtain the initial fast features and the initial slow features. The slowly changing physiological features are the initial slow features, and the rapidly changing physiological features are the rapidly changing physiological features. The initial fast features and the initial slow features can represent the feature information of different time dimensions.

[0112] The above downsampling ratio can be set according to actual conditions, and this embodiment does not specifically limit this.

[0113] The above-mentioned time series downsampling processing of the spatiotemporal features to obtain the initial fast features and the initial slow features may include using a preset fifth neural network model to perform time series downsampling processing on the spatiotemporal features to obtain the initial fast features and the initial slow features, but is not limited to this.

[0114] The fifth neural network model mentioned above can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0115] For example, the fifth neural network model may be at least one layer of convolutional neural network, but is not limited thereto.

[0116] The time length of the above-mentioned spatiotemporal feature is a first preset multiple of the time length of the above-mentioned initial slow feature.

[0117] The first preset multiple is greater than one.

[0118] The above-mentioned first preset multiple can be set according to actual conditions, and the embodiment of the present application does not make any specific limitation on this.

[0119] For example, the first preset multiple may be four times, that is, if the time length of the spatiotemporal feature is 128, then the time length of the initial slow feature is 32, but is not limited thereto.

[0120] The time length of the initial fast feature is a second preset multiple of the time length of the initial slow feature.

[0121] The second preset multiple is greater than one.

[0122] The above-mentioned second preset multiple can be set according to actual conditions, and the embodiment of the present application does not make any specific limitation on this.

[0123] For example, the second preset multiple may be twice, that is, if the time length of the initial slow feature is 32, then the time length of the initial fast feature is 64, but it is not limited thereto.

[0124] The number of channels of the above-mentioned spatiotemporal features is equal to the number of channels of the above-mentioned initial slow features.

[0125] For example, if the number of channels of the spatiotemporal feature is 64, the number of channels of the initial slow feature is also 64.

[0126] The number of channels of the spatiotemporal feature is a third preset multiple of the number of channels of the initial fast feature.

[0127] The third preset multiple is greater than one.

[0128] The third preset multiple mentioned above can be set according to actual conditions, and the embodiment of the present application does not make any specific limitation on this.

[0129] For example, the third preset multiple may be twice, that is, if the number of channels of the spatiotemporal feature is 64, the number of channels of the initial fast feature is 32, but the present invention is not limited thereto.

[0130] In one example, the above initial slow feature can be expressed as , is the time length of the above initial slow feature, , the above initial fast feature can be expressed as , is the time length of the initial fast feature, and the number of channels of the initial fast feature is compressed to half of the original number of channels, that is, , The number of channels representing spatiotemporal features.

[0131] In some implementations, the above-mentioned initial fast features are combined with a temporal difference module to obtain deep fast features, including:

[0132] Extracting the initial fast features through the temporal difference module and the pooling layer to obtain the first fast features;

[0133] Extracting the first fast feature through a temporal difference module and a pooling layer to obtain a second fast feature;

[0134] The second fast feature is extracted through the temporal difference module to obtain the deep fast feature.

[0135] In this implementation, a time difference module is introduced to perform feature extraction processing on the initial fast features. Figure 2 , Figure 2 "Temporal Difference Mamba" means the Timing Difference Module. Figure 2There are two processing branches as shown. The branch marked with "fast flow" is the processing branch of the initial fast feature, and the branch marked with "slow flow" is the processing branch of the initial slow feature. In the processing branch of the initial fast feature, first, the initial fast feature is extracted by the temporal difference module, and the spatial dimension of the feature is reduced by the pooling layer to obtain the first fast feature, so as to preliminarily capture the detail information and long-distance spatiotemporal dependency of the rPPG signal in the rapidly changing physiological features; then, the first fast feature is extracted by the temporal difference module, and the spatial dimension of the feature is reduced by the pooling layer to obtain the second fast feature, so as to further refine the detail information and long-distance spatiotemporal dependency of the rPPG signal in the rapidly changing physiological features; finally, the second fast feature is extracted by the temporal difference module to obtain the deep fast feature, so as to extract the detail information and long-distance spatiotemporal dependency of the more complex rPPG signal in the rapidly changing physiological features, thereby effectively capturing the long-distance spatiotemporal dependency in the rapidly changing physiological features, which is beneficial to improve the accuracy and reliability of the rPPG signal.

[0136] The pooling layer in the above feature extraction is arranged behind the above temporal difference module, which is used to reduce the spatial dimension of the feature.

[0137] The pooling layer in the above feature extraction may be a maximum pooling layer; or, the pooling layer in the above feature extraction may be an average pooling layer, but is not limited thereto.

[0138] For example, the pooling layer in the above feature extraction is 1 2 2 max pooling layers, but not limited to this.

[0139] In some implementations, the above-mentioned deep slow features are obtained based on the initial fast features and the initial slow features in combination with the time difference module, which may include:

[0140] The initial slow feature is extracted through the temporal difference module and the pooling layer to obtain the first slow feature;

[0141] Performing a lateral connection process on the first slow feature and the first fast feature to obtain a second slow feature;

[0142] The second slow feature is extracted through a temporal difference module and a pooling layer to obtain a third slow feature;

[0143] Performing a lateral connection process on the third slow feature and the second fast feature to obtain a fourth slow feature;

[0144] The fourth slow feature is extracted through the temporal difference module to obtain a deep slow feature.

[0145] In this embodiment, a temporal difference module is introduced to extract features from the initial slow features, and the feature extraction results of the initial slow features are fused with the feature extraction results of the initial fast features. Figure 2 , Figure 2 "Temporal Difference Mamba" means the Timing Difference Module. Figure 2 As shown in the figure, there are two processing branches, the branch marked with "fast flow" is the processing branch of the initial fast feature, and the branch marked with "slow flow" is the processing branch of the initial slow feature. In the processing branch of the initial fast feature, the first fast feature and the second fast feature are obtained by feature extraction; in the processing branch of the initial slow feature, firstly, the initial slow feature is extracted by the temporal difference module, and the spatial dimension of the feature is reduced by the pooling layer to obtain the first slow feature, so as to preliminarily capture the detailed information and long-distance spatiotemporal dependency of the rPPG signal in the slowly changing physiological features, and then the first fast feature is fused with the first slow feature by lateral connection to obtain the second slow feature, so as to realize the feature fusion of multi-scale time dimension; then, the second slow feature is extracted by the temporal difference module. The fast feature is used for feature extraction, and the spatial dimension of the feature is reduced through the pooling layer to obtain the third slow feature, so as to further refine the detail information and long-distance spatiotemporal dependency of the rPPG signal in the slowly changing physiological features. Then, the second fast feature is fused with the third slow feature through a lateral connection to obtain the fourth slow feature, so as to realize feature fusion of multi-scale time dimensions. Finally, the fourth slow feature is extracted through the temporal difference module to obtain the deep slow feature, so as to extract the more complex detail information and long-distance spatiotemporal dependency of the rPPG signal in the slowly changing physiological features, thereby effectively capturing the long-distance spatiotemporal dependency in the slowly changing physiological features, and realizing the effective fusion of physiological features of different time dimensions, reducing feature redundancy and ensuring feature accuracy.

[0146] The implementation method of the above-mentioned horizontal connection can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0147] For example, the above lateral connection can be achieved by convolution kernel size 3 1 1. Step size is 2 1 1. Fill to 1 0 0 convolution block to achieve, but not limited to this.

[0148] In some embodiments, reference Figure 2 , Figure 2In the “Temporal Difference Mamba” of , “Temporal Difference Convolution” corresponds to the temporal difference convolution layer, “Flatten” and the first “Layer Normalization” from top to bottom constitute the flattened normalization structure, “Bidirectional Mamba” represents the bidirectional state space structure, ", the second "layer normalization" and "shape reshaping" from top to bottom constitute the skip processing structure, and "channel attention" represents the attention layer;

[0149] The above-mentioned timing difference module may include:

[0150] The temporal difference convolution layer is used to perform temporal difference convolution processing on the input features of the temporal difference module to obtain a temporal difference feature map;

[0151] A flattening normalization structure is used to flatten the time series difference feature map into a one-dimensional feature sequence, and perform layer normalization on the one-dimensional feature sequence to obtain a layer-normalized one-dimensional feature sequence;

[0152] A bidirectional state space structure is used to process the layer-normalized one-dimensional feature sequence based on at least two state space models to obtain a first bidirectional time series sequence;

[0153] A jump processing structure, used for performing jump connection on the first bidirectional time series sequence and the one-dimensional feature sequence to obtain a second bidirectional time series sequence, and performing layer normalization processing on the second bidirectional time series sequence to obtain a second bidirectional time series sequence after layer normalization;

[0154] The attention layer is used to process the second bidirectional time series sequence after layer normalization based on the attention mechanism to obtain the output features of the time difference module.

[0155] In this embodiment, in order to capture the long-distance spatiotemporal dependencies in rapidly changing or slowly changing physiological characteristics to the greatest extent and improve the reliability and accuracy of the rPPG signal, a timing difference module is introduced. The timing difference module is configured with state space models in different timing directions. The state space model can well capture all long-distance spatiotemporal dependencies.

[0156] Specifically, in the temporal difference module, it mainly includes a temporal difference convolution layer, a flattened normalization structure, a bidirectional state space structure, a jump processing structure and an attention layer, wherein the input features of the temporal difference module are subjected to temporal difference convolution processing through the temporal difference convolution layer to extract a fine local temporal difference feature map based on subtle temporal skin color changes, thereby obtaining a temporal difference feature map; the temporal difference feature map is converted into a one-dimensional feature sequence through the flattened normalization structure, and the one-dimensional feature sequence is subjected to layer normalization processing to improve the generalization ability of the temporal difference module; the one-dimensional feature sequence after layer normalization is subjected to layer normalization processing based on at least two state space models through the bidirectional state space structure. The feature sequence is processed to capture the long-distance spatiotemporal dependency between frames, thereby obtaining a first bidirectional time series sequence; the first bidirectional time series sequence and the one-dimensional feature sequence are jump-connected through a jump processing structure to achieve multi-level feature fusion and improve the robustness of the rPPG feature representation, thereby obtaining a second bidirectional time series sequence, and then the second bidirectional time series sequence is layer-normalized to obtain a second bidirectional time series sequence after layer normalization, so as to normalize the second bidirectional time series sequence and convert its shape into the same shape as the time difference feature map; the channel representation of the second bidirectional time series sequence after layer normalization is enhanced through an attention layer to obtain the output features of the time difference module.

[0157] The above-mentioned performing temporal difference convolution processing on the input features of the temporal difference module to obtain the temporal difference feature map may include performing a convolution operation on the input features of the temporal difference module to obtain a convolution operation result, performing a temporal difference operation on the input features of the temporal difference module to obtain a temporal difference operation result, and obtaining the temporal difference feature map based on the convolution operation result and the temporal difference operation result.

[0158] For example, with a convolution kernel size of 3 3 Taking the temporal difference convolution of 3 as an example, the temporal difference feature map satisfies the following formula (1):

[0159] (1);

[0160] In formula (1), represents a temporal difference feature map; Represents the convolution operation; represents a temporal difference operation; is a learnable weight parameter; Represents the spatiotemporal location of the input feature or output feature map; Indicates the current frame time step (i.e. ) and its adjacent time steps (i.e. and ) The local spatiotemporal receptive field area of ​​the sampled convolution operation, Indicates the field of view All space-time locations in Represents two adjacent frame time steps (i.e. and ), It is mainly used to aggregate the time difference clues of two adjacent frame time steps, calculate the feature changes in two adjacent frame time steps, and combine the feature changes into the convolution output of the current time step by difference; is a hyperparameter, which is mainly used to adjust the weight of time difference information. The higher the hyperparameter, the higher the importance of time difference. When , the temporal difference convolution operation can be regarded as a normal three-dimensional convolution operation.

[0161] The flattened normalized structure may include a flattening layer and a normalization layer connected sequentially, but is not limited thereto.

[0162] The flattening layer of the above flattened normalization structure is used to flatten the temporal difference feature map into a one-dimensional feature sequence, for example, The spatiotemporal differential feature map is flattened to a size of One-dimensional feature sequence of .

[0163] The normalization layer of the flattened normalization structure is used to perform layer normalization processing on the one-dimensional feature sequence to obtain a one-dimensional feature sequence after layer normalization.

[0164] The normalization layer for flattening the normalized structure may be a layer normalization layer, but is not limited thereto.

[0165] The above-mentioned one-dimensional feature sequence after layer normalization is processed based on at least two state space models to obtain a first bidirectional time series sequence, which may include processing the one-dimensional feature sequence after layer normalization through a linear Gaussian state space model to obtain a first model feature, processing the one-dimensional feature sequence after layer normalization through a Mamba model to obtain a second model feature, and fusing the first model feature and the second model feature to obtain a first bidirectional time series sequence, but is not limited to this.

[0166] The above-mentioned jump processing structure may include a connector and a normalization layer connected in sequence, but is not limited thereto.

[0167] The connector of the jump processing structure is used to perform jump connection on the first bidirectional time series and the one-dimensional feature sequence to obtain a second bidirectional time series.

[0168] The above-mentioned skip connection may be an addition connection, but is not limited thereto.

[0169] The second bidirectional timing sequence satisfies the following formula (2):

[0170] (2);

[0171] In formula (2), represents a one-dimensional feature sequence, Representation layer normalization processing, represents the processing of bidirectional state-space structures, Represents the second bidirectional timing sequence.

[0172] The normalization layer of the above-mentioned jump processing structure is used to perform layer normalization processing on the second bidirectional time series sequence to obtain the second bidirectional time series sequence after layer normalization, so as to normalize the second bidirectional time series sequence and convert its shape into the same shape as the time series difference feature map.

[0173] The normalization layer of the skip processing structure may be a layer normalization layer, but is not limited thereto.

[0174] The above-mentioned attention mechanism may be a channel attention mechanism; or, the above-mentioned attention mechanism may be a self-attention mechanism, a multi-head attention mechanism, etc., but is not limited thereto.

[0175] In one example, the attention layer is used to process the second bidirectional time series sequence after layer normalization through a channel attention mechanism to obtain output features of a temporal difference module.

[0176] In some embodiments, reference Figure 2 , Figure 2 In the "Bidirectional Mamba" in , there are two "convolutions" and two "SSMs", one of which constitutes the first temporal state space layer, and the other "convolution" and "SSM" constitute the second temporal state space layer. ” and a “ ” constitutes a gated processing structure;

[0177] The above two-way state space structure may include:

[0178] A first linear layer is used to linearly map the one-dimensional feature sequence after layer normalization into two identical hidden states, where the two identical hidden states include a first state feature and a second state feature, and the time sequence directions of the first state feature and the second state feature are both the first direction;

[0179] An activation layer, used for processing the first state feature through a preset activation function to obtain the activated first state feature;

[0180] The first temporal state space layer is used to perform convolution processing on the second state feature to obtain a first convolution feature, and process the first convolution feature through a Mamba model to obtain a first temporal state feature;

[0181] The second temporal state space layer is used to perform a flipping process on the second state feature along the time direction to obtain a flipped second state feature, wherein the temporal direction of the flipped second state feature is a second direction, and the second direction is opposite to the first direction, and the flipped second state feature is convolved to obtain a second convolution feature, and the second convolution feature is processed through a Mamba model to obtain a second temporal state feature;

[0182] The gated processing structure is used to perform gated processing on the activated first state feature and the first timing state feature to obtain a first fusion feature, perform gated processing on the activated first state feature and the second timing state feature to obtain a second fusion feature, and add the first fusion feature and the second fusion feature to obtain a first bidirectional timing sequence.

[0183] In this implementation, in a bidirectional state space structure, long-range spatiotemporal dependencies between frames are captured through two temporal directions, forward and backward.

[0184] Specifically, first, the one-dimensional feature sequence after layer normalization is linearly mapped into two identical hidden states through a first linear layer, and the two identical hidden states include a first state feature and a second state feature, and the time series direction of the first state feature and the second state feature is the first direction. Then, the first state feature is delinearized based on a preset activation function through an activation layer to obtain an activated first state feature; at the same time, the second state feature is convoluted to obtain a first convolution feature, and the first convolution feature is processed by a Mamba model to capture the long-distance spatiotemporal dependency in the first direction to obtain a first time series state feature, and the second state feature is flipped along the time direction to obtain a second state feature with a second direction in the time series direction, and the second direction is opposite to the first direction, and then the second state feature with a second direction in the time series direction is convoluted to obtain a second convolution feature, and the second convolution feature is processed by a Mamba model to capture the long-distance spatiotemporal dependency in the second direction to obtain a second time series state feature. Afterwards, the activated first state feature and the first timing state feature are gated through a gating processing structure to obtain a first fusion feature, the activated first state feature and the second timing state feature are gated to obtain a second fusion feature, and the first fusion feature and the second fusion feature are added to obtain a first bidirectional timing sequence.

[0185] The timing directions of the first state characteristic and the second state characteristic are both the first direction.

[0186] The timing direction of the second state feature after the flip is the second direction.

[0187] The second direction is opposite to the first direction.

[0188] The first direction and the second direction can be set according to actual conditions, and this embodiment does not specifically limit this.

[0189] For example, if the first direction is forward, the second direction is backward; or, if the first direction is backward, the second direction is forward.

[0190] The number of channels of the first state feature and the second state feature can be expanded by a preset expansion factor to facilitate the calculation of the state space model.

[0191] The above expansion factor can be set according to actual conditions, and this embodiment does not specifically limit this.

[0192] The activation function in the above activation layer can be set according to actual conditions, and this embodiment does not make any specific limitation on this.

[0193] For example, the activation function in the above activation layer may be a SiLU activation function, but is not limited thereto.

[0194] The convolution processing in the first temporal state space layer and the second temporal state space layer may include implementing the convolution processing through one-dimensional convolution, but is not limited thereto.

[0195] The above-mentioned one-dimensional convolution can be cascaded with a preset activation function, and the activation function can be set according to actual conditions. This embodiment does not specifically limit this. For example, the activation function cascaded by the above-mentioned one-dimensional convolution can be a SiLU activation function, but is not limited to this.

[0196] The above Mamba model is a state-space model, which is used to model dynamic systems through state-time representation. Specifically, the state-space model uses hidden states The one-dimensional function Mapping to another function , this hidden state can usually be described by a continuous time-invariant ordinary differential equation system, as shown in the following formula (3):

[0197] , (3);

[0198] In formula (3), represents the output of a continuous time invariant ordinary differential equation system, is the first learnable parameter, is the second learnable parameter, is the third learnable parameter.

[0199] In order to apply the continuous time invariant ordinary differential equation system to the extraction of rPPG signals, this embodiment adopts a discretization method to convert the parameters and parameters Specifically, using the preset sampling period time step Parameters and parameters Discretization is performed using the zero-order hold discretization method, as shown in the following formula (4):

[0200] , (4);

[0201] In formula (4), represents the matrix exponential operation, is the preset identity matrix, is the first parameter after discretization, is the second parameter after discretization, is the sampling period time step.

[0202] Through the above-mentioned zero-order hold discretization method, this embodiment can transform the continuous time invariant ordinary differential equation system into a linear recursive model, so as to apply the continuous time invariant ordinary differential equation system to the extraction of rPPG signals, and the input of the linear recursive model can be defined as a time step. Based on this, according to the above-mentioned time step, the hidden state of the state space model is re-expressed as the following formula (5):

[0203] , (5).

[0204] The above formula (5) can be used to implement the state space model operation of the first convolution feature and the second convolution feature. It can be seen that compared with the traditional spatial state model where system parameters do not change with time and input, this embodiment provides a state space model that depends on input. The selected scanning mechanism makes the parameters , and With input is derived from the changes in .

[0205] The above-mentioned implementation of the state space model operation of the first convolution feature and the second convolution feature may include implementing the operation and parallel training of the state space model of the first convolution feature and the second convolution feature in a global convolution manner. Specifically, the global convolution satisfies the following formula (6):

[0206] , (6);

[0207] In formula (6), represents the convolution kernel, Representation sequence Length, represents the convolution operation, Represents the result of the convolution operation.

[0208] The gate control processing structure may include a first multiplier, a second multiplier and an adder, but is not limited thereto.

[0209] The first multiplier is used to perform gate processing on the activated first state feature and the first timing state feature to obtain a first fusion feature.

[0210] The second multiplier is used to perform gate processing on the activated first state feature and the second timing state feature to obtain a second fusion feature.

[0211] The adder is used to add the first fusion feature and the second fusion feature to obtain a first bidirectional time series.

[0212] A second linear layer may be arranged behind the above-mentioned gate control processing structure, and the second linear layer may be used to perform linear transformation processing on the first bidirectional timing sequence.

[0213] In one example, in the above bidirectional state space structure, first, the input sequence will be mapped to the hidden state and hidden state , hidden state and hidden state All are hidden states of the forward time sequence, and the hidden states are flipped along the time direction , get the hidden state of the backward sequence Then, for the hidden state , through the SiLU activation function to the hidden state Processing to obtain the hidden state after activation ; For the hidden state of the forward sequence and the hidden state of the backward sequence , both have: First, a one-dimensional convolution using a cascaded SiLU activation function is applied to the hidden state , that is, for the hidden state Perform convolution processing, the hidden state after convolution processing is linearly mapped to parameters ,parameter and time step ; Then, using the preset sampling period time step Parameters and parameters Discretize and get the discretized parameters and the discretized parameters ; Afterwards, use the parameter and parameters Combined with the above formula (5), the state space model is operated to obtain the state characteristics of the corresponding direction time series, namely the first time series state characteristics and the second time series state characteristics. Finally, the first time series state characteristics are combined with the activated hidden state Gating is performed to combine the second temporal state feature with the activated hidden state Gating is performed to obtain two gating results, and the two gating results are added to obtain a first bidirectional time series sequence, and the first bidirectional time series sequence is output through a second linear layer.

[0214] In some embodiments, obtaining the physiological signal according to the deep fast feature and the deep slow feature may include:

[0215] The deep slow features are upsampled to obtain upsampled deep slow features, and the scale of the deep fast features in the time dimension is the same as the scale of the upsampled deep slow features in the time dimension;

[0216] According to the deep fast features and the upsampled deep slow features, the fast-slow fusion deep features are obtained;

[0217] Based on the fast and slow fusion deep features and combined with the preset predictor, the physiological signal is obtained.

[0218] In this implementation, first, the deep slow features are upsampled to obtain the upsampled deep slow features, aiming to make the scale of the deep fast features in the time dimension the same as the scale of the upsampled deep slow features in the time dimension; then, based on the deep fast features and the upsampled deep slow features, the fast-slow fusion deep features are obtained, aiming to further fuse the physiological features of different time dimensions; finally, the physiological signal prediction is performed based on the fast-slow fusion deep features and the preset predictor to obtain the required physiological signal, i.e., the rPPG signal, thereby realizing the extraction and detection of the rPPG signal.

[0219] The scales of the deep fast features and the deep slow features in the time dimension can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the scales of the deep fast features and the deep slow features in the time dimension can be 64, but are not limited to this.

[0220] In an example, the above upsampled deep slow features can be expressed as , the above deep fast features can be expressed as , but not limited to this.

[0221] The fast-slow fused deep features are obtained according to the deep fast features and the upsampled deep slow features, which may include directly concatenating the deep fast features and the upsampled deep slow features into the fast-slow fused deep features.

[0222] Alternatively, the fast-slow fused deep features obtained according to the deep fast features and the upsampled deep slow features may include merging the deep fast features and the upsampled deep slow features in the feature channel dimension to obtain the fast-slow fused deep features, but is not limited to this.

[0223] The above-mentioned method of obtaining the physiological signal based on the fast and slow fusion deep features in combination with a preset predictor may include inputting the fast and slow fusion deep features into a preset predictor, processing the fast and slow fusion deep features through the predictor, and obtaining the rPPG signal, but is not limited to this.

[0224] The above predictor can be set according to actual conditions, and this embodiment does not specifically limit this.

[0225] The above-mentioned processing of the fast and slow fusion deep features by the predictor to obtain the rPPG signal may include performing temporal upsampling, spatial pooling and one-dimensional signal mapping processing on the fast and slow fusion deep features in sequence by the predictor to obtain the rPPG signal, but is not limited to this.

[0226] In some embodiments, the above method further comprises:

[0227] According to the physiological signal, the parameters of the timing difference module are updated.

[0228] In this implementation, in order to improve the performance and generalization ability of the temporal difference module, the parameters of the temporal difference module are updated based on the physiological signal.

[0229] The above-mentioned updating of the parameters of the timing difference module may include calculating the difference value between the physiological signal and the expected signal corresponding to the physiological signal through a preset loss function, and updating the parameters of the timing difference module based on the difference value in combination with back propagation, but is not limited to this.

[0230] The above loss function can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0231] For example, the above loss function may be a negative Pearson loss function, which can ensure that the time patterns of the predicted rPPG signal and the corresponding expected signal are consistent, which is beneficial to the extraction of the rPPG signal, but is not limited to this.

[0232] In order to facilitate the understanding of the physiological signal processing method of the present application, the actual application scenario of the physiological signal processing method of the present application is used as an example. Figure 2 The specific process of extracting rPPG signals from facial long sequence video data in this application is as follows: steps S201-S208.

[0233] S201, photographing and processing the user's face through a camera to obtain face video data, performing face detection on the first frame initial face image in the face video data through a multi-task convolutional neural network model to obtain the face area in the first frame initial face image, and taking the coordinates of the face area in the first frame initial face image as the target coordinates; for each frame initial face image, cropping out the area with coordinate information as the target coordinates in the initial face image as the face image; traversing each frame initial face image to obtain each frame face image; constructing a long face sequence video data through each frame face image, the long face sequence video data includes 128 frames face images, the frame height and frame width of each frame face image are both 128, and each frame face image contains three primary color channels of red, green and blue.

[0234] S202, performing differential normalization processing on the long sequence video data of the face to obtain differential video frames of the long sequence video data of the face , , , .

[0235] S203, convolution processing is performed on the differential video frame through a shallow backbone network to obtain the spatiotemporal features of the long facial video sequence data. The shallow backbone network includes three sequentially connected convolution modules. The input of the first convolution module is the differential video frame, the output of the first convolution module is used as the input of the second convolution module, the output of the second convolution module is used as the input of the last convolution module, and the output of the last convolution module is the spatiotemporal features of the long facial video sequence data. , , , .

[0236] Specifically, the first convolution module and the last convolution module both include sequentially connected convolution blocks, pooling layers, normalization layers, and activation function layers. The second convolution module includes sequentially connected convolution blocks, normalization layers, and activation function layers. The convolution kernel size of the convolution block of the first convolution module is 1. 5 5. The convolution kernel size of the convolution block of the second convolution module is 3 3 3. The convolution kernel size of the convolution block of the third convolution module is 3 3 3; The convolution block is used to perform convolution processing on the input data of the convolution module, the pooling layer is used to perform pooling processing on the output data of the convolution block, the normalization layer is used to normalize the output data of the pooling layer, and the activation function layer is used to delinearize the output data of the normalization layer to obtain the output data of the convolution module.

[0237] S204, based on a preset downsampling ratio, performing time-series downsampling processing on the spatiotemporal features to obtain an initial fast feature and the initial slow feature , , .

[0238] S205, extracting the initial fast feature through the temporal difference module and the pooling layer to obtain the first fast feature; extracting the first fast feature through the temporal difference module and the pooling layer to obtain the second fast feature; extracting the second fast feature through the temporal difference module to obtain the deep fast feature.

[0239] And, extract the initial slow feature through the time difference module and the pooling layer to obtain the first slow feature; perform horizontal connection processing on the first slow feature and the first fast feature to obtain the second slow feature; extract the second slow feature through the time difference module and the pooling layer to obtain the third slow feature; perform horizontal connection processing on the third slow feature and the second fast feature to obtain the fourth slow feature; extract the fourth slow feature through the time difference module to obtain the deep slow feature. Among them, the horizontal connection is through the convolution kernel size of 3 1 1. Step size is 2 1 1. Fill to 1 0 0 convolution block to achieve this.

[0240] In the temporal difference module, first, the input features of the temporal difference module are processed by the temporal difference convolution layer to obtain the temporal difference feature map, as shown in the above formula (1); then, the normalized structure with shape The spatiotemporal differential feature map is flattened to a size of A one-dimensional feature sequence is obtained, and the one-dimensional feature sequence is layer-normalized to obtain a one-dimensional feature sequence after layer normalization; then, the one-dimensional feature sequence after layer normalization is processed based on at least two state space models through a bidirectional state space structure to obtain a first bidirectional time series sequence.

[0241] Specifically, in the bidirectional state space structure, first, the input sequence will be mapped to the hidden state and hidden state , hidden state and hidden state All are hidden states of the forward time sequence, and the hidden states are flipped along the time direction , get the hidden state of the backward sequence Then, for the hidden state , through the SiLU activation function to the hidden state Processing to obtain the hidden state after activation ; For the hidden state of the forward sequence and the hidden state of the backward sequence , both have: First, a one-dimensional convolution using a cascaded SiLU activation function is applied to the hidden state , that is, for the hidden state Perform convolution processing, the hidden state after convolution processing is linearly mapped to parameters ,parameter and time step ; Then, using the preset sampling period time step Parameters and parameters Discretize and get the discretized parameters and the discretized parameters ; Afterwards, use the parameter and parameters Combined with the above formula (5), the state space model is operated to obtain the state characteristics of the corresponding direction time series, namely the first time series state characteristics and the second time series state characteristics. Finally, the first time series state characteristics are combined with the activated hidden state Gating is performed to combine the second temporal state feature with the activated hidden state Gating is performed to obtain two gating results, and the two gating results are added to obtain a first bidirectional timing sequence, and the first bidirectional timing sequence is linearly transformed and then output.

[0242] After obtaining the first bidirectional time series sequence, the first bidirectional time series sequence and the one-dimensional feature sequence are jump-connected through a jump processing structure to obtain a second bidirectional time series sequence, and the second bidirectional time series sequence is layer-normalized to obtain a second bidirectional time series sequence after layer normalization, so as to normalize the second bidirectional time series sequence and convert its shape into the same shape as the time series difference feature map.

[0243] Finally, the second bidirectional time series sequence after layer normalization is processed by the attention layer based on the channel attention mechanism to obtain the output features of the temporal difference module.

[0244] S206, upsampling the deep slow feature to obtain the upsampled deep slow feature, the scale of the deep fast feature in the time dimension is the same as the scale of the upsampled deep slow feature in the time dimension, and the scales are both 64.

[0245] S207, merging the deep fast features and the upsampled deep slow features in the feature channel dimension to obtain a fast-slow fused deep feature.

[0246] S208, inputting the fast and slow fusion deep features into a preset predictor, and performing temporal upsampling, spatial pooling and one-dimensional signal mapping processing on the fast and slow fusion deep features in sequence through the predictor to obtain rPPG signals.

[0247] In summary, the embodiment of the present application processes the long-sequence video data to obtain its spatiotemporal characteristics, and obtains the initial fast characteristics and the initial slow characteristics based on the spatiotemporal characteristics, so as to effectively and accurately capture the slowly changing physiological characteristics and the rapidly changing physiological characteristics in the spatiotemporal characteristics. The slowly changing physiological characteristics are the initial slow characteristics, and the rapidly changing physiological characteristics are the rapidly changing physiological characteristics. The initial fast characteristics and the initial slow characteristics represent different time dimensions respectively.

[0248] Furthermore, the embodiment of the present application can effectively capture the long-distance spatiotemporal dependencies (especially the long-distance spatiotemporal dependencies between the forward and backward time series) in the rapidly changing physiological features by extracting the initial fast features through a temporal difference module provided with a bidirectional Mamba model;

[0249] Furthermore, the embodiment of the present application extracts features of the initial slow features by setting up a timing difference module with a bidirectional Mamba model, which can not only effectively capture the long-distance spatiotemporal dependencies in the slowly changing physiological features (especially the long-distance spatiotemporal dependencies between the forward timing and the backward timing), but also realize the effective fusion of physiological features in different time dimensions, reduce feature redundancy, and ensure feature accuracy.

[0250] Furthermore, the embodiments of the present application achieve accurate detection and extraction of rPPG signals by fusing deep fast features and deep slow features that represent long-distance spatiotemporal dependencies, and extracting and detecting rPPG signals based on the fused features.

[0251] Furthermore, the embodiment of the present application can effectively reduce the number of parameters in the rPPG signal extraction process and maintain a relatively low computational complexity, which is conducive to lightweight deployment in resource-constrained mobile devices. 128 Taking 128 as an example, the embodiment of the present application can reduce the number of parameters in the rPPG signal extraction process to 560,000, and the number of multiplications and additions is only 4.73 billion times, which can achieve lightweight deployment.

[0252] In summary, the embodiments of the present application effectively improve the accuracy, reliability and robustness of rPPG signals.

[0253] In addition, refer to Figure 3 The present application also provides a physiological signal processing device, which includes:

[0254] An acquisition module 301 is used to acquire facial long sequence video data;

[0255] The first processing module 302 is used to process the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data;

[0256] The second processing module 303 is used to obtain an initial fast feature and an initial slow feature according to the spatiotemporal feature; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively;

[0257] The third processing module 304 is used to obtain deep fast features based on the initial fast features in combination with the time series difference module; wherein the time series difference module is used to extract features based on a preset state space model;

[0258] The fourth processing module 305 is used to obtain a deep slow feature according to the initial fast feature and the initial slow feature in combination with a time difference module;

[0259] The fifth processing module 306 is used to obtain a physiological signal according to the deep fast feature and the deep slow feature.

[0260] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0261] Finally, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to implement the above-mentioned physiological signal processing method when executed by the processor.

[0262] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0263] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0264] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0265] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0266] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution system, device or apparatus. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution system, device or apparatus.

[0267] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0268] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0269] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0270] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0271] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A physiological signal processing method, characterized in that: The following steps are involved: Obtain facial long sequence video data; Processing the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data; According to the spatiotemporal features, an initial fast feature and an initial slow feature are obtained; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively; According to the initial fast features, combined with the time series difference module, a deep fast feature is obtained; wherein the time series difference module is used to extract features based on a preset state space model; According to the initial fast feature and the initial slow feature, combined with the time series difference module, a deep slow feature is obtained; Obtaining a physiological signal according to the deep fast feature and the deep slow feature; Wherein, obtaining the initial fast feature and the initial slow feature according to the spatiotemporal feature includes: Based on a preset downsampling ratio, the spatiotemporal features are subjected to time series downsampling processing to obtain the initial fast features and the initial slow features.

2. The physiological signal processing method according to claim 1, characterized in that: The processing of the long sequence video data of the face to obtain the spatiotemporal features of the long sequence video data of the face includes: Performing differential normalization processing on the long facial video sequence data to obtain differential video frames of the long facial video sequence data; The differential video frame is subjected to at least one convolution process to obtain the spatiotemporal features of the facial long sequence video data.

3. The physiological signal processing method according to claim 1, characterized in that: The method of obtaining a deep fast feature based on the initial fast feature and combining the time difference module includes: Extracting the initial fast feature through the temporal difference module and the pooling layer to obtain a first fast feature; Extracting the first fast feature through the temporal difference module and the pooling layer to obtain a second fast feature; The second fast feature is extracted by the temporal difference module to obtain the deep fast feature.

4. The physiological signal processing method according to claim 3, characterized in that: The step of obtaining a deep slow feature based on the initial fast feature and the initial slow feature in combination with the time series difference module includes: Extracting the initial slow feature through the temporal difference module and the pooling layer to obtain a first slow feature; Performing a lateral connection process on the first slow feature and the first fast feature to obtain a second slow feature; Extracting the second slow feature through the temporal difference module and the pooling layer to obtain a third slow feature; Performing a lateral connection process on the third slow feature and the second fast feature to obtain a fourth slow feature; The fourth slow feature is extracted by the temporal difference module to obtain the deep slow feature.

5. The physiological signal processing method according to claim 1, characterized in that: The timing difference module includes: A temporal difference convolution layer, used for performing temporal difference convolution processing on the input features of the temporal difference module to obtain a temporal difference feature map; A flattening normalization structure is used to flatten the time series difference feature map into a one-dimensional feature sequence, and perform layer normalization processing on the one-dimensional feature sequence to obtain a layer-normalized one-dimensional feature sequence; A bidirectional state space structure, used for processing the one-dimensional feature sequence after the layer normalization based on at least two state space models to obtain a first bidirectional time series sequence; a jump processing structure, used for performing jump connection on the first bidirectional time series sequence and the one-dimensional feature sequence to obtain a second bidirectional time series sequence, and performing layer normalization processing on the second bidirectional time series sequence to obtain a second bidirectional time series sequence after layer normalization; The attention layer is used to process the second bidirectional time series sequence after normalization of the layer based on the attention mechanism to obtain the output features of the time difference module.

6. The physiological signal processing method according to claim 5, characterized in that: The bidirectional state space structure comprises: A first linear layer, used for linearly mapping the one-dimensional feature sequence after normalization of the layer into two identical hidden states, wherein the two identical hidden states include a first state feature and a second state feature, and the time sequence directions of the first state feature and the second state feature are both in a first direction; An activation layer, used for processing the first state feature through a preset activation function to obtain an activated first state feature; A first temporal state space layer is used to perform convolution processing on the second state feature to obtain a first convolution feature, and process the first convolution feature through a Mamba model to obtain a first temporal state feature; A second temporal state space layer is used to perform a flipping process on the second state feature along the time direction to obtain a flipped second state feature, wherein the temporal direction of the flipped second state feature is a second direction, and the second direction is opposite to the first direction, and convolution processing is performed on the flipped second state feature to obtain a second convolution feature, and the second convolution feature is processed through a Mamba model to obtain a second temporal state feature; A gated processing structure is used to perform gated processing on the activated first state feature and the first timing state feature to obtain a first fusion feature, perform gated processing on the activated first state feature and the second timing state feature to obtain a second fusion feature, and add the first fusion feature and the second fusion feature to obtain the first bidirectional timing sequence.

7. The physiological signal processing method according to claim 1, characterized in that: The step of obtaining a physiological signal according to the deep fast feature and the deep slow feature comprises: Upsampling the deep slow feature to obtain the upsampled deep slow feature; wherein the scale of the deep fast feature in the time dimension is the same as the scale of the upsampled deep slow feature in the time dimension; Obtaining a fast-slow fusion deep feature according to the deep fast feature and the upsampled deep slow feature; The physiological signal is obtained according to the fast and slow fusion deep features combined with a preset predictor.

8. A physiological signal processing device, characterized in that: include: An acquisition module is used to acquire facial long sequence video data; A first processing module is used to process the facial long sequence video data to obtain the spatiotemporal features of the facial long sequence video data; A second processing module is used to obtain an initial fast feature and an initial slow feature according to the spatiotemporal feature; wherein the initial fast feature and the initial slow feature are features representing different time dimensions respectively; A third processing module is used to obtain deep fast features based on the initial fast features in combination with a time series difference module; wherein the time series difference module is used to extract features based on a preset state space model; A fourth processing module, configured to obtain a deep slow feature according to the initial fast feature and the initial slow feature in combination with the time series difference module; a fifth processing module, configured to obtain a physiological signal according to the deep fast feature and the deep slow feature; Wherein, obtaining the initial fast feature and the initial slow feature according to the spatiotemporal feature includes: Based on a preset downsampling ratio, the spatiotemporal features are subjected to time series downsampling processing to obtain the initial fast features and the initial slow features.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to implement a physiological signal processing method as claimed in any one of claims 1 to 7 when executed by the processor.

Images