A method and device for cross-period identity recognition based on PPG signals based on domain adaptation

Abstract

This invention discloses a method and device for cross-period identity recognition using PPG signals based on domain adaptation. The method first acquires a user's PPG signal by capturing fingertip video from a smart mobile device. The captured PPG signal is then used as input to a deep neural network to complete the identity recognition task. To ensure the accuracy of PPG signal recognition in cross-period scenarios, a domain-adversarial neural network (DANN) is designed based on the deep neural network and combined with domain adaptation. This invention leverages the convenience and speed of mobile devices and the high accuracy of neural networks to provide users with a safe and convenient user experience.

Description

Technical Field

[0001] The present invention belongs to the field of PPG signal technology, and in particular relates to a method and device for cross-period identity recognition of PPG signals based on domain adaptation. Background Art

[0002] In today's world, with the continuous advancement of information technology, the security of information and property for individuals, businesses, and even government agencies is becoming increasingly prominent. As the primary barrier to protecting against these risks, identity recognition technology faces higher requirements and more challenges. To meet the demands of the current information age, photoplethysmographic (PPG)-based identity recognition technology has become a research hotspot in the field, thanks to its advantages of being difficult to forge and theft and requiring liveness verification.

[0003] As a more contemporary identification method, PPG signals offer greater concealment and are difficult to forge or steal than previous generations of identification technologies. They require liveness verification for identification, resulting in superior security. Compared to contemporary ECG signals, they are easier to collect and do not require high-end, expensive equipment. This offers broader application prospects and a high demand for use, making them a research hotspot in recent years. In terms of application scenarios, PPG signals can address patient identification and personalized diagnosis in smart healthcare, secure payment and property protection in the financial sector, and workplace requirements such as employee recruitment and clocking in. But at the same time, the current identity recognition task based on PPG signals also has defects: as a data-driven task, the performance of the identity recognition task based on PPG signals will be highly dependent on the input data. The inherent changes in the user's PPG signals in different periods will inevitably lead to a decrease in the accuracy of identity recognition, and the system availability will be greatly reduced. For this reason, it is necessary to design a method that can meet the needs of cross-period PPG signal identity recognition. This method can extract the common features of the user's PPG signals in different periods, and the extracted common features can remain stable in different periods to meet the needs of long-term authentication, and can also ensure the distinguishability of the input features of the identity recognition system and the accuracy of the recognition results. Summary of the Invention

[0004] Technical issues:

[0005] This invention aims to provide a method for cross-period identity recognition based on PPG signals. This method addresses the problem of existing deep neural networks, which suffer from reduced accuracy and significantly reduced system usability due to inherent variations in a user's PPG signals over time. Based on deep neural networks, a DANN network was designed with domain adaptation to improve the accuracy of identity recognition in cross-period scenarios.

[0006] Technical solution:

[0007] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and device for cross-period identity recognition based on PPG signals based on domain adaptation, comprising the following steps:

[0008] S1: Collect fingertip videos through the smartphone camera and upload them to the cloud server to complete the initial data collection;

[0009] S2: Extract the PPG signal with the best signal quality based on the fingertip video. First, filter it with a second-order Butterworth filter of 0.5-20Hz to reduce the noise interference of motion artifacts and power frequency interference while retaining the main frequency range of the PPG signal. Then standardize it with Z-score to make the mean of the PPG signal become 0 and the standard deviation become 1. The standardized PPG signal is windowed with the main peak as the starting point. The signal window size is 100 and 1s is selected as the signal window duration. That is, if the sampling frequency is 100Hz, the PPG signal contains 100 data points, and the PPG signal segments that do not meet 1s will be discarded.

[0010] S3: The PPG signal is used as the input of a deep neural network (DNN) for training. The DNN includes a one-dimensional convolutional neural network (1D CNN), a bidirectional gated logic unit (BiGRU), a self-attention mechanism (Self-Attention), and a fully connected layer.

[0011] S4: The PPG signal identity recognition task based on deep neural network DNN has the problem of reduced identity recognition accuracy due to the intrinsic changes of PPG signals in cross-period scenarios. In order to better learn the common features of PPG signals in different periods, based on the deep neural network DNN in step S3, a domain adversarial neural network DANN is built in combination with domain adaptation; the DANN network consists of four parts, namely, feature extractor G f , label classifier G y , domain discriminator G d and gradient reversal layer GRL; the DNN network in step S3 is partially reused as the feature extractor of the DANN network except the fully connected layer, and a fully connected layer is added on the basis of the feature extractor to design the label classifier G y and domain discriminator G d ;

[0012] S5: The identity recognition task is completed through the DANN network, and the final identity recognition result is transmitted back to the user App terminal for continuing downstream identity recognition related tasks.

[0013] Furthermore, the PPG signal with the best signal quality performance was obtained through comparison: in the color space channel, the PPG signal quality performance of the R, G, and B channels of the RGB space, the Y channel of the YIQ, and the V channel of the HSV space were compared; in the signal extraction area, the image of the input fingertip video was divided into multiple ROI video sub-blocks and the PPG signal quality performance was compared on the corresponding video sub-blocks respectively; the signal quality performance was measured by amplitude change AC and signal-to-noise ratio SNR.

[0014]

[0015] where X j represents the PPG signal segment of the jth signal window. The signal window size is set to 30. n represents the total number of current PPG signal windows. The last signal window whose length does not meet 30 will be discarded.

[0016] SNR = 10log(P s / P n ),

[0017] Among them, P s and P n Represent the power P of the signal and noise respectively. For a signal x(n) with a signal length of n, the calculation formula of the power P is

[0018] Furthermore, the deep neural network DNN includes a one-dimensional convolutional neural network layer 1D CNN, a bidirectional gated logic unit BiGRU, a self-attention mechanism Self-Attention and a fully connected layer; the DNN input PPG signal dimension is 100*1, the number of convolution layers of the one-dimensional convolutional neural network layer is 1, the convolution layer kernel size is 3*1, the number of convolution kernels is 64, the Dropout is 20%, the number of bidirectional GRU layers is 1, the GRU hidden state is 256, the input of the self-attention mechanism is 128*2, the input and output of the fully connected layer are 256*1 / number of authenticated persons*1, the weight decay term is 0.001, and 0.01 is used for L2 regularization.

[0019] Furthermore, combined with the domain-adaptive domain adversarial neural network DANN, a total of feature extractors G d , label classifier G y , domain discriminator G d and gradient reversal layer GRL; reuse the network content of the DNN network except the fully connected layer as the feature extractor of the DANN network, and design the label classifier G through the fully connected layer respectively. y , domain discriminator G d; The label classifier consists of 2 layers of fully connected FC layers. The input / output of the first FC layer is 512*1 / 128*1 respectively, followed by a Dropout layer with a random inactivation rate of 50%. The input / output of the second FC layer is 128*1 / number of classified people*1 respectively; the domain discriminator also consists of 2 layers of FC layers. The input / output of the first FC layer is 56*1 / 256*1 respectively, and the input / output of the second FC layer is 256*1 / 2*1 respectively; the gradient reversal layer is used to control the reversal strength. Its parameter is controlled by λ, and the size settings include 1, 1.5, and 2.

[0020] Furthermore, combined with the domain-adaptive domain adversarial neural network DANN, the objective function includes the label classification loss L ce and domain classification loss L dann , label classification loss L ce Calculated by cross entropy loss function;

[0021]

[0022] Where p(x i ) represents the probability that the true label is the category (0 or 1); q(x i ) represents the probability of the model predicting the category, and m represents the total number of all categories.

[0023]

[0024] Where x s and x t Represent the data samples of the source domain and the target domain respectively, p s (x) and p t (x) represents the marginal probability distribution of source domain and target domain data samples, θ f Represents the feature extractor G f Parameter, θ d Representative domain discriminator G d Parameter, E represents the expectation of the corresponding project.

[0025] A domain-adaptive PPG signal cross-period identity recognition device is characterized in that it includes a smartphone and a cloud server, wherein the camera of the smartphone completes the fingertip video recording task in claim 1 and uploads the collected fingertip video to the cloud server; the cloud server implements the contents of steps S2-S5 in claims 1-5.

[0026] Beneficial effects:

[0027] By combining domain adaptation technology, the PPG signal of the current period is regarded as the source domain, and the PPG signal of the future or another period is regarded as the target domain. The feature maps of the PPG signals of different periods are aligned, and then the optimal decision boundary is re-divided. This can effectively alleviate the problem of decreased accuracy in cross-period scene identity recognition. The model combined with domain adaptation can not only perform well in the environment or scene of the current period, but also maintain comparable performance in the environment or scene of the future or another period. Moreover, because the domain adaptation technology fully considers the changes in the characteristics of the PPG signals in different periods, the model can better capture the characteristics of the PPG signals in different periods. Even for data that has not appeared in the training process, the model's recognition performance will be better. Secondly, the current public datasets for cross-period identity recognition only contain PPG signals from two different periods, while domain adaptation can expand the identity recognition task to data from two or more periods. Unlike the traditional method of manually extracting features, it can avoid complex feature engineering construction and has better recognition performance than existing neural network methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 DANN network structure diagram in the present invention.

[0029] Figure 2 Schematic diagram of fingertip video acquisition based on a mobile phone in the present invention.

[0030] Figure 3 Schematic diagram of video sub-block division of fingertip video in the present invention

[0031] Figure 4 AC heatmaps of PPG signals of different video sub-blocks in the present invention.

[0032] Figure 5 Heatmap of SNR of PPG signals of different video sub-blocks in the present invention.

[0033] Figure 6 Visualization result diagram of the DANN network ONNX model of the present invention.

[0034] Figure 7 t-SNE diagram of PPG signals of 31 users of the DANN network in different periods of time in the present invention.

[0035] Figure 8 Schematic diagram of the mobile phone Android App interface in the present invention. Specific implementation methods

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and implementation examples. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Example 1

[0039] This embodiment provides a method for cross-period identity recognition based on domain adaptation of PPG signals, and the DANN neural network structure used is as follows: Figure 1 As shown, the following steps are included:

[0040] Step 1: Collect fingertip video through the smartphone camera, such as Figure 2 As shown, the fingertip video is uploaded to the cloud server to complete the initial data collection;

[0041] Step 2: Complete the PPG signal extraction of the G channel on the specific video sub-block of the fingertip video. The video sub-block division information is as follows: Figure 3 As shown in the figure, the signal was filtered using a second-order Butterworth filter with a frequency of 0.5 to 20 Hz and then normalized using the Z-score. The normalized PPG signal was windowed with the main peak as the starting point, and the size of the signal window was 100. Figure 4 and Figure 5 The AC and SNR heatmap distributions of each video sub-block are shown separately. Based on this, the PPG signal is extracted only on the union of video sub-blocks 9 and 10, resulting in the best PPG signal quality.

[0042] Step 3: Use the PPG signal as input to train a deep neural network (DNN). The deep neural network (DNN) includes a one-dimensional convolutional neural network (1D CNN), a bidirectional gated logic unit (BiGRU), a self-attention mechanism (Self-Attention), and a fully connected layer.

[0043] Step 4: Combine domain adaptation to build a domain adversarial neural network DANN. Figure 6 The visualization results of the DANN network ONNX model are shown. The network can effectively learn the common features of PPG signals at different stages. The DANN network consists of four parts: feature extractor G f , label classifier G y , domain discriminator G d and gradient reversal layer GRL; the DNN network in step 3 is partially reused as the feature extractor of the DANN network except the fully connected layer, and a fully connected layer is added on the basis of the feature extractor to design the label classifier Gy and domain discriminator G d The t-SNE results of the 31 users in the network at different times are as follows Figure 7 As shown in the figure, the feature distribution of the same user has the same color, and the different shapes represent the PPG signal features of different periods.

[0044] Step 5: The identity recognition task is completed through the DANN network, and the final identity recognition result is transmitted back to the user's App terminal. The relevant interface of the App terminal is as follows Figure 8 , used to continue downstream identity recognition related tasks.

[0045] The exemplary embodiments described above merely represent several embodiments of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person of ordinary skill in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for cross-period identity recognition based on PPG signals based on domain adaptation, characterized in that: The following steps are involved: S1: Collect fingertip videos through the smartphone camera and upload them to the cloud server to complete the initial data collection; S2: Extract the PPG signal with the best signal quality based on the fingertip video. First, filter it with a second-order Butterworth filter of 0.5-20Hz to reduce the noise interference of motion artifacts and power frequency interference while retaining the frequency range of the PPG signal. Then standardize it with Z-score to make the mean of the PPG signal become 0 and the standard deviation become 1. The standardized PPG signal is windowed with the main peak as the starting point. The signal window size is 100 and 1s is selected as the signal window duration. That is, if the sampling frequency is 100Hz, the PPG signal contains 100 data points, and the PPG signal segments that do not meet 1s will be discarded. The PPG signal with optimal signal quality was determined by comparing the R, G, and B channels of the RGB space, the Y channel of the YIQ (Yiqu) space, and the V channel of the HSV space. In the signal extraction area, the input fingertip video image was divided into multiple ROI video sub-blocks and the PPG signal quality performance was compared in each corresponding video sub-block. Signal quality was measured by amplitude change (AC) and signal-to-noise ratio (SNR). where X j represents the PPG signal segment of the jth signal window. The signal window size is set to 30. N represents the total number of current PPG signal windows. The last signal window whose length does not meet 30 will be discarded. SNR=10log(P s / P n ), Among them, P s and P n Represent the power P of the signal and noise respectively. For a signal x(n) with a signal length of n, the calculation formula of the power P is S3: The PPG signal is used as the input for training the deep neural network DNN, which includes a one-dimensional convolutional neural network 1DCNN, a bidirectional gated logic unit BiGRU, a self-attention mechanism Self-Attention and a fully connected layer; S4: The PPG signal identity recognition task based on deep neural network DNN has the problem of reduced identity recognition accuracy due to the intrinsic changes of PPG signals in cross-period scenarios. In order to better learn the common features of PPG signals in different periods, based on the deep neural network DNN in step S3, a domain adversarial neural network DANN is built in combination with domain adaptation; the DANN network consists of four parts, namely, feature extractor G f , label classifier G y , domain discriminator G d and gradient reversal layer GRL; the DNN network in step S3 is partially reused as the feature extractor of the DANN network except the fully connected layer, and a fully connected layer is added on the basis of the feature extractor to design the label classifier G y and domain discriminator G d ; Combined with the domain-adaptive domain adversarial neural network DANN, the objective function includes the label classification loss L ce and domain classification loss L dann , label classification loss L ce Calculated by cross entropy loss function; Where p(x i ) represents the probability that the true label is category i, which is 0 or 1; q(x i ) represents the probability of the model predicting category i, and m represents the total number of all categories; Where x s and x t Represent the data samples of the source domain and the target domain respectively, p s (x) and p t (x) represents the marginal probability distribution of source domain and target domain data samples, θ f Represents the feature extractor G f Parameter, θ d Representative domain discriminator G d parameter, represents the expectations of the corresponding items; S5: The identity recognition task is completed through the DANN network, and the final identity recognition result is transmitted back to the user App terminal for continuing downstream identity recognition related tasks.

2. The method for cross-period identity recognition based on PPG signals based on domain adaptation according to claim 1, characterized in that: The deep neural network DNN includes a one-dimensional convolutional neural network layer 1D CNN, a bidirectional gated logic unit BiGRU, a self-attention mechanism Self-Attention and a fully connected layer; the DNN input PPG signal dimension is 100*1, the number of convolution layers of the one-dimensional convolutional neural network layer is 1, the convolution layer kernel size is 3*1, the number of convolution kernels is 64, the Dropout is 20%, the number of bidirectional GRU layers is 1, the GRU hidden state is 256, the input of the self-attention mechanism is 128*2, the input / output of the fully connected layer is 256*1 / number of authenticated persons*1 respectively, the weight decay term is 0.001, and the L2 regularization parameter is 0.

01.

3. The method for cross-period identity recognition based on PPG signals based on domain adaptation according to claim 1, characterized in that: Combined with the domain adaptive domain adversarial neural network DANN, a total of feature extractor G f , label classifier G y , domain discriminator G d and gradient reversal layer GRL; reuse the network content of the DNN network except the fully connected layer as the feature extractor of the DANN network, and design the label classifier G through the fully connected layer respectively. y and domain discriminator G d ; The label classifier consists of 2 layers of fully connected FC layers. The input / output of the first FC layer is 512*1 / 128*1 respectively, followed by a Dropout layer with a random inactivation rate of 50%. The input / output of the second FC layer is 128*1 / number of classified people*1 respectively; the domain discriminator also consists of 2 layers of FC layers. The input / output of the first FC layer is 56*1 / 256*1 respectively, and the input / output of the second FC layer is 256*1 / 2*1 respectively; the gradient reversal layer is used to control the reversal strength. Its parameter is controlled by λ, and the size settings include 1, 1.5, and 2.

4. A domain-adaptive PPG signal cross-period identity recognition device, characterized in that: It includes a smart phone and a cloud server, wherein the camera of the smart phone completes the fingertip video recording task in claim 1 and uploads the collected fingertip video to the cloud server; the cloud server implements the contents of steps S2-S5 in claims 1-3.

Images