Non-contact heart rate monitoring method based on palm video and related equipment

Pulse waves are extracted through the convolutional neural network of palm video data and the generation adversarial network, combined with the long-term memory network and peak detection algorithm, the technical problem of contactless heart rate monitoring is solved, and the monitoring effect is achieved with convenient, accurate and privacy protection.

CN118948237APending Publication Date: 2024-11-15XIAN EASY PALM TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411034113.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The prior art central rate monitoring methods are difficult to achieve contactlessness, expensive and inconvenient equipment, easy to cause discomfort in wearing equipment, and privacy and environmental interference problems are caused by heart rate monitoring based on face video.

Method used

The contactless heart rate monitoring method based on palm video is adopted to extract pulse wave data through convolutional neural network and generative adversarial network, and heart rate estimation is performed by combining long-term memory network and peak detection algorithm to achieve contactless and convenient heart rate monitoring.

Benefits of technology

It realizes contactless heart rate monitoring, enhances user privacy protection, reduces equipment costs, improves monitoring accuracy and portability, and avoids interference from lighting and posture changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118948237A_ABST
    Figure CN118948237A_ABST
Patent Text Reader

Abstract

The invention relates to a biomedical signal processing technology, and discloses a palm video-based non-contact heart rate monitoring method and related equipment, original palm video data is loaded and preprocessed to obtain complete palm area data; analyzing complete palm area data, performing network training, and extracting pulse wave data; the extracted pulse wave data are analyzed and processed, heart rate estimation is carried out, and heart rate monitoring is carried out according to the heart rate estimation result. Pulse wave signals are extracted through palm videos, complete non-contact measurement is achieved, and the heart rate can be monitored only by placing the palm in front of the camera by a user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to biomedical signal processing technology, and in particular to a palm video-based contactless heart rate monitoring method and related equipment. Background Art

[0002] Traditional heart rate measurement methods mainly rely on electrocardiograms or wearable heart rate monitoring devices. Although these methods are accurate, they require professional equipment and are not convenient for daily use. Electrocardiogram monitoring requires electrodes to directly contact the skin, and the equipment is expensive and not suitable for daily portable use. Wearable heart rate monitoring devices, such as smart bracelets and heart rate belts, are convenient but need to be worn for a long time, which can easily cause discomfort and compliance problems for users.

[0003] With the development of computer vision and signal processing technology, it has become possible to analyze biometric signals through video. This technology uses ordinary cameras to collect video data and extracts physiological signals by analyzing small changes in the video. It has the advantages of being contactless and convenient, and is particularly suitable for home health monitoring and telemedicine.

[0004] At present, methods for extracting pulse wave (PPG) signals and estimating heart rate based on face videos have been widely studied. These methods obtain pulse wave signals by detecting slight color changes of facial skin. However, heart rate monitoring based on face videos has certain privacy issues. The collection and processing of face videos may involve the exposure of personal identity information, which can easily cause users to worry about privacy leakage. In addition, the collection of face videos has high requirements for lighting conditions and head posture, and is easily disturbed by the external environment. Summary of the invention

[0005] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a contactless heart rate monitoring method based on palm video and related equipment to solve the technical problem of how to achieve contactless heart rate measurement in the prior art.

[0006] The present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a contactless heart rate monitoring method based on palm video, comprising:

[0008] Load and preprocess the original palm video data to obtain complete palm area data;

[0009] Conduct convolutional neural network training on the complete palm area data, and introduce generative adversarial network based on convolutional neural network training to obtain pulse wave data;

[0010] The pulse wave data is analyzed and processed to obtain frequency characteristics, the heart rate is estimated through a peak detection algorithm based on the frequency characteristics, and the heart rate is monitored according to the heart rate estimation result.

[0011] Preferably, the specific process of loading and preprocessing the original palm video data to obtain complete palm area data is as follows:

[0012] The original palm video data is subjected to palm area detection and tracking using machine learning algorithms and then cropped to obtain the complete palm area;

[0013] The video frame data and pulse wave labels of the complete palm area are aligned and normalized.

[0014] Furthermore, the specific process of using machine learning algorithms to detect and track the palm area of ​​the original palm video data and cropping the complete palm area is as follows:

[0015] Locate the key points of the palm to obtain the palm area, and select multiple key points in the center of the palm area to obtain the coordinates of the palm center point and the width and height of the bounding box;

[0016] The coordinates of the cropping area are calculated by the coordinates of the center point of the palm and the width and height of the bounding box, and the complete palm area is obtained according to the coordinates of the cropping area.

[0017] Preferably, in network training, spatial feature extraction is performed based on a convolutional neural network, and time series features are extracted in combination with a long short-term memory network, and a generative adversarial network is introduced to generate false pulse wave data and false palm videos using random noise, and the forged pulse wave data and the real pulse wave data are obtained by discrimination through a discriminator, and the network parameters are adjusted, and the forged pulse wave data and the real pulse wave data as well as the real data are sent to the network for training, and a self-supervised learning method is introduced to obtain the pulse wave data by training the original pulse wave data for data masking or data reconstruction tasks.

[0018] Furthermore, the specific process of introducing the generative adversarial network GAN is as follows:

[0019] Use a convolutional neural network as a generator, input random noise z, and output the generated pulse wave data G (Z) At the same time, the discriminator is constructed. In the last layer, the Sigmoid function is used to map the output to the interval [0, 1], which indicates the probability of judging it as real data. Its input is the real pulse wave data x or the forged pulse wave data G. (Z) , the output is judged to be true probability D (x) or D(G (Z) ).

[0020] Furthermore, two strategies are used for network training. The first strategy is to fix the discriminator D, update the parameters of the generator G, calculate the gradient of the generator, and use the Mamba technology to optimize the training gradient update strategy. The second strategy is to fix the generator G, update the parameters of the discriminator D, calculate the gradient of the discriminator with respect to the real data and the gradient of the discriminator with respect to the generated data, and finally add the two gradients to obtain the total gradient of the discriminator, and then use the optimization algorithm to update the parameters of the discriminator.

[0021] Preferably, in the analysis and processing of the extracted pulse wave data, the extracted pulse wave data is trended to remove interference caused by noise and fluctuations, and the frequency information related to the heart rate is retained; and the filtered frequency information is fast Fourier transformed to extract frequency features, and the heart rate is estimated through a peak detection algorithm.

[0022] In a second aspect, the present invention provides a non-contact heart rate monitoring system based on palm video, comprising:

[0023] The loading and preprocessing module is used to load and preprocess the original palm video data to obtain complete palm area data;

[0024] The network training module is used to train the convolutional neural network on the complete palm area data, and introduce the generative adversarial network based on the convolutional neural network training to obtain the pulse wave data;

[0025] The data monitoring module is used to analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate based on the frequency characteristics through a peak detection algorithm, and monitor the heart rate according to the heart rate estimation result.

[0026] In a third aspect, the present invention provides a mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the contactless heart rate monitoring method based on palm video as described above is implemented.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the contactless heart rate monitoring method based on palm video as described above.

[0028] Compared with the prior art, the present invention has the following beneficial technical effects:

[0029] The present invention provides a contactless heart rate monitoring method based on palm video, which loads and preprocesses the original palm video data to obtain complete palm area data; analyzes the complete palm area data, performs network training, and extracts pulse wave data; analyzes and processes the extracted pulse wave data, estimates the heart rate, and monitors the heart rate based on the heart rate estimation result. By extracting the pulse wave signal through the palm video, completely contactless measurement is achieved, and the user only needs to place the palm in front of the camera to monitor the heart rate.

[0030] Furthermore, the original palm video data is loaded and preprocessed to obtain complete palm area data, which can effectively extract pulse wave information from the complete palm area data and eliminate other interfering backgrounds.

[0031] Furthermore, in network training, spatial feature extraction is performed based on convolutional neural networks, and time series features are extracted in combination with long short-term memory networks. A feature extraction module and signal enhancement algorithm based on convolutional neural networks (CNN) are used to effectively amplify PPG signals and suppress noise, thereby improving the signal-to-noise ratio of PPG signals. The accuracy of heart rate estimation is further improved through algorithms such as adaptive filters, Fourier transforms, high-pass filters, and peak detection.

[0032] Furthermore, in the feature extraction stage, convolutional neural networks are used to extract spatial features, and long short-term memory networks are combined to extract time series features. In terms of data enhancement, the generator of the generative adversarial network generates realistic pulse wave data, and the discriminator distinguishes between real and generated data, thereby enhancing the diversity and magnitude of training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a contactless heart rate monitoring method based on palm video in an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the structure of a contactless heart rate monitoring system based on palm video in an embodiment of the present invention;

[0035] Figure 3 It is a model framework diagram in an embodiment of the present invention;

[0036] In the figure: 1- loading and preprocessing module; 2- network training module; 3- data monitoring module. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0039] The purpose of the present invention is to provide a contactless heart rate monitoring method based on palm video and related equipment to solve the technical problem of how to achieve contactless heart rate measurement in the prior art.

[0040] Example 1

[0041] See also Figure 1 and Figure 2 In one embodiment of the present invention, a non-contact heart rate monitoring method based on palm video is provided, comprising:

[0042] Step 1, loading and preprocessing the original palm video data to obtain complete palm area data;

[0043] Specifically, the specific process in step 1 is as follows:

[0044] Step 11, using a machine learning algorithm to detect and track the original palm video data and cropping the original palm area to obtain a complete palm area;

[0045] Among them, since the input video in the real scene not only contains the palm but also may contain some complex background and other interference factors, the palm detection model based on the MediaPipe library is used to crop the central area of ​​the palm. The resulting palm area not only contains rich pulse wave information, but also effectively eliminates other interfering backgrounds.

[0046] The process of step 11 is as follows:

[0047] The complete palm area is obtained by locating the key points of the palm, and then the key points in the center of the four palm areas are selected to obtain the coordinates of the palm center point and the width and height of the bounding box: Center x , Center y , Box w , Box h Finally, the coordinates of the clipping area are calculated from the values ​​of these coordinates: X min , X max , Y min , Ymax .

[0048] X min =Center x -Box w / 2

[0049] X max =Center x +Box w / 2

[0050] Y min =Center y -Box h / 2

[0051] Y max =Center y +Box h / 2

[0052] Step 12: align and normalize the video frame data and pulse wave labels of the complete palm area.

[0053] Specifically, while the original video frames are cropped and preprocessed, the real pulse wave signal also needs to be normalized, and then the original video frames and real labels are aligned on the timeline, and all video frame data and label data are divided into non-overlapping data segments for subsequent network training to ensure that each video frame segment corresponds to its real pulse wave label to avoid data intersection and mismatch, and also align the video frames with the real label data on the timeline.

[0054] Step 2: Perform convolutional neural network training on the complete palm area data, and introduce a generative adversarial network based on the convolutional neural network training to obtain pulse wave data;

[0055] Specifically, in network training, spatial feature extraction is performed based on a convolutional neural network, and time series features are extracted in combination with a long short-term memory network. A generative adversarial network (GAN) is introduced, and false pulse wave data and false palm videos are generated using random noise. The discriminator is used to discriminate between the forged pulse wave data and the real pulse wave data, and the network parameters are adjusted. The forged pulse wave data, the real pulse wave data and the real data are sent to the network for training. A self-supervised learning method is introduced to obtain the pulse wave data by training the original pulse wave data for data masking or data reconstruction tasks.

[0056] Among them, because the pulse wave signal has a strong periodicity and conforms to the characteristics of the time series, we perform a differential operation on the original video frame and analyze the differences between different video frame sequences to capture dynamic change information. The calculation method is as follows:

[0057] ΔI(t)=I(t)-I(t-1)

[0058] Wherein ΔI(t) represents the frame difference between the previous frame and the next frame, I(t) represents the current frame, and I(t-1) represents the previous frame.

[0059] Dynamic frame differences are used when calculating frame differences. Specifically, a fixed sliding window step size is set, which effectively solves the problem of incomplete feature information due to similar frame differences. At the same time, in order to capture more feature information, the video is sampled on the time axis, so that local features at different time scales can be obtained, which helps the model learn long-term dependency information and dynamic changes.

[0060] Specifically, in order to improve the performance of the model, the present invention combines the latest technologies Mamba, Generative Adversarial Network (GAN), self-supervised learning and multimodal data fusion. In the feature extraction stage, a convolutional neural network (CNN) is used to extract spatial features, and a long short-term memory network (LSTM) is combined to extract time series features. In terms of data enhancement, the generator of the Generative Adversarial Network (GAN) generates realistic pulse wave data, and distinguishes between real and generated data through a discriminator, thereby enhancing the diversity and magnitude of training data.

[0061] The specific formula is as follows:

[0062] Feature spatial =CNN(I(t))

[0063] Feature temporal =LSTM(ΔI(t))

[0064] Among them, Feature spatial Represents the extracted spatial features, Feature temporal Represents the extracted time series features.

[0065] In the process of network training, in order to solve the problem of insufficient and shortage of data and real pulse wave data, the present invention introduces the generative adversarial network GAN, uses the convolutional neural network as the generator, inputs random noise z, and outputs the generated pulse wave data G (Z) At the same time, a discriminator is constructed. The structure is similar to that of the generator. In the last layer, the Sigmoid function is used to map the output to the interval [0, 1], indicating the probability of judging it as real data. Its input is the real pulse wave data x or the generated pulse wave data G (Z) , the output is judged to be true probability D (x) or D(G (z)). During training, for the generator, the discriminator judges the generated data as real, and its objective function is:

[0066]

[0067] For the discriminator, the objective function is to accurately distinguish between real and generated data:

[0068]

[0069] Among them, p data (x) represents the distribution of real data, p z (z) represents the distribution of noise.

[0070] Specifically, two strategies are used for network training. The first strategy is to fix the discriminator D, update the parameters of the generator G, calculate the gradient of the generator, and use the Mamba technology to optimize the training gradient update strategy. The second strategy is to fix the generator G, update the parameters of the discriminator D, calculate the gradient of the discriminator with respect to the real data and the gradient of the discriminator with respect to the generated data, and finally add the two gradients to obtain the total gradient of the discriminator, and then use the optimization algorithm to update the parameters of the discriminator.

[0071] By continuously training the generator, a large amount of realistic pulse wave data is generated. Assuming there are M samples, it can be expressed as:

[0072]

[0073] These generated data can be used together with the original real pulse wave data to train the original pulse wave extraction and heart rate estimation model. During the training process, the real data and generated data are mixed in a certain proportion. This can increase the diversity and quantity of data, thereby improving the generalization ability of the original model.

[0074] During the training process, the self-supervised learning method is introduced. The main implementation method is to train the original pulse wave data with data masking or data reconstruction tasks, so that the model can learn meaningful feature representation without manual labeling. For example, a part of the pulse wave signal is randomly masked, and the model is allowed to learn to predict the masked part, thereby mining the potential patterns in the data.

[0075] The calculation formula is as follows:

[0076] y masked =Mask(y)

[0077] Among them, y is the pulse wave data. The final loss function uses the mean absolute error to evaluate the performance of the model. The calculation formula is as follows:

[0078]

[0079] Among them, y i is the true value, is the predicted value of the model and N is the number of samples.

[0080] The network is adjusted inversely by calculating the loss between the true value and the predicted value, so that the network continues to converge. Finally, the features after average pooling are flattened into a one-dimensional vector as the final prediction result. The best network training result will be saved as a pre-training model for subsequent real-time prediction tasks.

[0081] Step 3: Analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate through a peak detection algorithm based on the frequency characteristics, and monitor the heart rate based on the heart rate estimation result.

[0082] Specifically, in the analysis and processing of the extracted pulse wave data, the extracted pulse wave data is trended, the interference caused by noise and fluctuations is removed, and the frequency information related to the heart rate is retained; and the filtered frequency information is fast Fourier transformed to extract the frequency characteristics, and the heart rate is estimated through the peak detection algorithm.

[0083] Here is the formula for implementation:

[0084] y filtered =y-LPF (y)

[0085] Perform high-pass filtering on the original signal to remove some noise interference and retain the frequency information related to the heart rate.

[0086]

[0087] Perform fast Fourier transform (FFT) on the filtered signal to extract the frequency features. At the same time, find the frequency peak and calculate the corresponding heart rate.

[0088] HeartRate=f peak ×60

[0089] Among them, f peak is the peak frequency.

[0090] In this embodiment, heart rate variability (HRV) is also important information for measuring the regulation of the human cardiovascular system, an important basis for judging cardiovascular diseases, and can predict sudden cardiac death and arrhythmia events. The pulse wave information obtained through the above steps can be further used to obtain some important indicators for measuring heart rate variability, including SDNN (reflecting the total tension of sympathetic and parasympathetic nerves), RMSSD (reflecting the tension of sympathetic nerves), LF (low-frequency power) and HF (high-frequency power). By calculating the values ​​of these indicators, changes in human heart activity can be measured, and some sudden diseases can also be warned. The following are formulas for calculating these indicators:

[0091]

[0092] Where N represents the total number of RR intervals, i represents the i-th RR interval, which is the time interval between adjacent heartbeats. is the mean of the RR intervals. SDNN is the standard deviation of the RR intervals, which is used to measure the overall volatility of the RR intervals and reflects the overall level of heart rate variability.

[0093]

[0094] Where N is the total number of RR intervals, i represents the i-th RR interval, RR i+1 Represents the i+1th RR interval. RMSSD is the root mean square value of the difference between adjacent RR intervals, which is used to measure the short-term changes in heart rate variability. A higher RMSSD value generally indicates higher heart health and better autonomic nervous function.

[0095]

[0096] Among them, P (f) The power spectrum density is LF power, which represents the power in the low-frequency range (0.04-0.15 Hz), reflecting the activity regulated by the sympathetic and parasympathetic nerves. The HF power represents the power in the high-frequency range (0.15-0.4 Hz), reflecting the parasympathetic nerve activity, especially the vagus nerve activity related to breathing.

[0097] Ultimately, by uploading data to the cloud for analysis and storage, health status can be monitored in real time and feedback and suggestions can be provided, thereby realizing health monitoring and early warning functions for users.

[0098] In this embodiment, after the above steps of data loading, data preprocessing, pulse wave extraction, heart rate estimation, network training, etc., the network finally trains the model to the cloud server and encrypts the data to ensure data security and protect user privacy. When performing subsequent prediction tasks, you only need to upload the captured video or the palm video captured by the real-time camera, and then the network can perform heart rate prediction and health monitoring tasks by loading the pre-trained model on the cloud.

[0099] In summary, this embodiment provides a contactless heart rate monitoring method based on palm video, which loads and preprocesses the original palm video data to obtain complete palm area data; analyzes the complete palm area data, performs network training, and extracts pulse wave data; analyzes and processes the extracted pulse wave data, estimates the heart rate, and monitors the heart rate based on the heart rate estimation result. By extracting the pulse wave signal through the palm video, completely contactless measurement is achieved, and the user only needs to place the palm in front of the camera to monitor the heart rate.

[0100] This embodiment extracts the pulse wave signal through the palm video, which does not involve the user's facial information, fundamentally avoids the leakage of personal identity information, and greatly protects the user's privacy. At the same time, data encryption and anonymization processing are used to further ensure the security and privacy protection of user data. In this embodiment, the color change of the palm area is more obvious, the stability of signal extraction is higher, and it is not easily affected by changes in light and posture. By extracting the pulse wave signal through the palm video, a more stable and accurate PPG signal can be obtained, and the accuracy of heart rate estimation can be improved. The feature extraction module and signal enhancement algorithm based on the convolutional neural network (CNN) are adopted to effectively amplify the PPG signal and suppress noise, thereby improving the signal-to-noise ratio of the PPG signal. The accuracy of heart rate estimation is further improved by algorithms such as adaptive filters, Fourier transforms, high-pass filters, and peak detection. The health monitoring module of this embodiment performs data analysis and storage through a cloud server, realizing remote health monitoring and big data analysis. Users can monitor their heart rate anytime and anywhere through the Internet, which is convenient for medical institutions to remotely monitor and manage the user's physiological condition, and has important application value.

[0101] Example 2

[0102] according to Figure 3 As shown, this embodiment also provides a non-contact heart rate monitoring system based on palm video, including a loading and preprocessing module 1, a network training module 2 and a data monitoring module 3;

[0103] The loading and preprocessing module 1 is used to load and preprocess the original palm video data to obtain complete palm area data;

[0104] Network training module 2 is used to perform convolutional neural network training on the complete palm area data, and introduce a generative adversarial network based on the convolutional neural network training to obtain pulse wave data;

[0105] The data monitoring module 3 is used to analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate based on the frequency characteristics through a peak detection algorithm, and monitor the heart rate according to the heart rate estimation result.

[0106] Example 3

[0107] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, such as a contactless heart rate monitoring program based on palm video.

[0108] When the processor executes the computer program, the steps of implementing the above-mentioned contactless heart rate monitoring method based on palm video are, for example:

[0109] Load and preprocess the original palm video data to obtain complete palm area data;

[0110] Conduct convolutional neural network training on the complete palm area data, and introduce generative adversarial network based on convolutional neural network training to obtain pulse wave data;

[0111] The pulse wave data is analyzed and processed to obtain frequency characteristics, the heart rate is estimated through a peak detection algorithm based on the frequency characteristics, and the heart rate is monitored according to the heart rate estimation result.

[0112] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized, for example:

[0113] The loading and preprocessing module 1 is used to load and preprocess the original palm video data to obtain complete palm area data;

[0114] Network training module 2 is used to perform convolutional neural network training on the complete palm area data, and introduce a generative adversarial network based on the convolutional neural network training to obtain pulse wave data;

[0115] The data monitoring module 3 is used to analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate based on the frequency characteristics through a peak detection algorithm, and monitor the heart rate according to the heart rate estimation result.

[0116] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the mobile terminal.

[0117] For example, the computer program may be divided into a loading and preprocessing module 1, a network training module 2, and a data monitoring module 3, and the specific functions of each module are as follows:

[0118] The loading and preprocessing module 1 is used to load and preprocess the original palm video data to obtain complete palm area data;

[0119] Network training module 2 is used to perform convolutional neural network training on the complete palm area data, and introduce a generative adversarial network based on the convolutional neural network training to obtain pulse wave data;

[0120] The data monitoring module 3 is used to analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate based on the frequency characteristics through a peak detection algorithm, and monitor the heart rate according to the heart rate estimation result.

[0121] The mobile terminal may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The mobile terminal may include, but is not limited to, a processor and a memory.

[0122] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the mobile terminal, and uses various interfaces and lines to connect various parts of the entire mobile terminal.

[0123] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the mobile terminal by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0124] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0125] Example 4

[0126] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the contactless heart rate monitoring method based on palm video are implemented.

[0127] If the module / unit integrated in the mobile terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0128] Based on this understanding, the present invention implements all or part of the processes in the above method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned non-contact heart rate monitoring method based on palm video can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0129] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0130] It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A non-contact heart rate monitoring method based on palm video, characterized in that: include: Load and preprocess the original palm video data to obtain complete palm area data; Conduct convolutional neural network training on the complete palm area data, and introduce generative adversarial network based on convolutional neural network training to obtain pulse wave data; The pulse wave data is analyzed and processed to obtain frequency characteristics, the heart rate is estimated through a peak detection algorithm based on the frequency characteristics, and the heart rate is monitored according to the heart rate estimation result.

2. The method for non-contact heart rate monitoring based on palm video according to claim 1, characterized in that: The specific process of loading and preprocessing the original palm video data to obtain complete palm area data is as follows: The original palm video data is subjected to palm area detection and tracking using machine learning algorithms and then cropped to obtain the complete palm area; The video frame data and pulse wave labels of the complete palm area are aligned and normalized.

3. The method for non-contact heart rate monitoring based on palm video according to claim 2, characterized in that: The specific process of using a machine learning algorithm to detect and track the palm area of ​​the original palm video data and cropping the complete palm area is as follows: Locate the key points of the palm to obtain the palm area, and select multiple key points in the center of the palm area to obtain the coordinates of the palm center point and the width and height of the bounding box; The coordinates of the cropping area are calculated by the coordinates of the center point of the palm and the width and height of the bounding box, and the complete palm area is obtained according to the coordinates of the cropping area.

4. The method for non-contact heart rate monitoring based on palm video according to claim 1, characterized in that: In the network training, spatial feature extraction is performed based on convolutional neural network, and time series features are extracted in combination with long short-term memory network. Generative adversarial network (GAN) is introduced, and false pulse wave data and false palm videos are generated by random noise. The discriminator is used to distinguish the forged pulse wave data and the real pulse wave data, and the network parameters are adjusted. The forged pulse wave data, the real pulse wave data and the real data are sent to the network for training. The self-supervised learning method is introduced to obtain the pulse wave data by training the original pulse wave data for data masking or data reconstruction tasks.

5. The method for non-contact heart rate monitoring based on palm video according to claim 4, characterized in that: The specific process of introducing the generative adversarial network is as follows: Use a convolutional neural network as a generator, input random noise z, and output the generated pulse wave data G (Z) , and construct a discriminator at the same time. In the last layer, the Sigmoid function is used to map the output to the interval [0, 1], indicating the probability of judging it as real data; Its input is the real pulse wave data x or the forged pulse wave data G (Z) , the output is judged to be true probability D (x) or D(G (Z) ).

6. The method for non-contact heart rate monitoring based on palm video according to claim 5, characterized in that: The network is trained using two strategies. The first strategy is to fix the discriminator D, update the parameters of the generator G, calculate the gradient of the generator, and use the Mamba technology to optimize the training gradient update strategy. The second strategy is to fix the generator G, update the parameters of the discriminator D, calculate the gradient of the discriminator with respect to the real data and the gradient of the discriminator with respect to the generated data, and finally add the two gradients to obtain the total gradient of the discriminator, and then use the optimization algorithm to update the parameters of the discriminator.

7. The method for non-contact heart rate monitoring based on palm video according to claim 1, characterized in that: In the process of analyzing and processing the extracted pulse wave data, the extracted pulse wave data is trended, interference caused by noise and fluctuations is removed, and frequency information related to the heart rate is retained; The filtered frequency information is subjected to fast Fourier transform to extract frequency features, and the heart rate is estimated through the peak detection algorithm.

8. A non-contact heart rate monitoring system based on palm video, characterized in that: include: A loading and preprocessing module (1) is used to load and preprocess the original palm video data to obtain complete palm area data; A network training module (2) is used to perform convolutional neural network training on the complete palm area data, and introduce a generative adversarial network based on the convolutional neural network training to obtain pulse wave data; The data monitoring module (3) is used to analyze and process the pulse wave data to obtain frequency characteristics, estimate the heart rate based on the frequency characteristics through a peak detection algorithm, and monitor the heart rate according to the heart rate estimation result.

9. A mobile terminal, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the contactless heart rate monitoring method based on palm video as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the contactless heart rate monitoring method based on palm video as described in any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Non-contact blood pressure detection method and system based on deep learning

    CN120336833A

  • Intelligent electrocardiogram abnormity prediction method and system based on video

    CN120678442A

  • Video-based intelligent method and system for electrocardio abnormality prediction

    CN120678442B