A method and device for heart rate detection

Abstract

This application discloses a heart rate detection method and apparatus. The method involves determining a first motion index of the user during the recording of a video including the user; then, based on the first motion index, determining at least one first video segment included in the video, in which the user's motion amplitude is less than a first threshold corresponding to the first motion index; and finally, determining the user's heart rate based on the first video segment. Specifically, within the first video segment, the user remains stationary or the user's motion amplitude is small. Correspondingly, the same area in each frame of the first video segment often includes the same part of the user's body. In this case, the accuracy of determining the user's heart rate through the first video segment is high. In other words, the solution provided by the embodiments of this application can improve the accuracy of heart rate detection.

Description

Technical Field

[0001] This application relates to the field of terminal equipment technology, specifically to a heart rate detection method and device. Background Technology

[0002] As living standards improve, people are paying more and more attention to their own and their families' health. Heart rate, as an important indicator for assessing human health, is also receiving increasing attention. To meet this demand, some devices, such as mobile phones and watches, now offer heart rate monitoring functions, allowing users to monitor their heart rate using these devices.

[0003] Currently, when terminal devices detect a user's heart rate, they typically use the following technology: First, they acquire a video of the user's face over a period of time. Then, based on multiple frames of images including the user's face in the video, they determine the photoplethysmograph (PPG) signal of the facial skin. Finally, they determine the user's heart rate based on the PPG signal.

[0004] However, heart rate detection using the above techniques usually suffers from low accuracy. Summary of the Invention

[0005] To address the issue of low accuracy in existing heart rate detection technologies, this application provides a heart rate detection method and apparatus.

[0006] In a first aspect, embodiments of this application disclose a heart rate detection method, including:

[0007] Determine the user's first motion index during the recording of a video including the user;

[0008] Based on the first motion index, at least one first video segment is determined to be included in the video, in which the user's motion amplitude is less than a first threshold corresponding to the first motion index;

[0009] The user's heart rate is determined based on the first video segment.

[0010] In this process, because the user's movement amplitude in the first video segment is less than the first threshold corresponding to the first motion index, the user remains stationary or exhibits minimal movement within the first video segment. Therefore, the above steps for determining the user's heart rate using the first video segment are highly accurate.

[0011] In one optional design, the first motion index includes at least one of the following: the displacement of the user's head, the angle change of the user's head, the displacement of m feature points of the user, the displacement of n grids, and the distance change between the grids;

[0012] Where m and n are positive integers, and the grid is the grid obtained by dividing the user's contour.

[0013] In one optional design, determining the user's heart rate based on the first video segment includes:

[0014] Based on the images included in the first video segment, determine the heart rate spectrum curve corresponding to the first video segment;

[0015] If the first video segment includes at least two elements, determine the weight of the first video segment;

[0016] The user's heart rate is determined based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment;

[0017] If the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

[0018] In the above steps, if the first video segment includes at least two segments, the user's heart rate can be determined according to the weight of each first video segment, thereby improving the accuracy of determining the user's heart rate based on each first video segment.

[0019] In one optional design, determining the weight of the first video segment includes:

[0020] The weight of the first video segment is determined based on its duration.

[0021] Alternatively, the weight of the first video segment can be determined based on the user's body parts included in the first video segment.

[0022] In one optional design, determining the heart rate spectrum curve corresponding to the first video segment includes:

[0023] Based on the photoplethysmography (PPG) signal of the same region in each frame of the first video segment, a first curve of the first video segment is determined. The horizontal axis of the first curve is time, and the vertical axis is the PPG signal. The same region in each frame includes the user.

[0024] By applying a bandpass filter to the first curve, a second curve that has undergone bandpass filtering is obtained.

[0025] The heart rate spectrum curve corresponding to the first video segment is determined by performing a fast Fourier transform on the second curve.

[0026] Secondly, embodiments of this application disclose a heart rate detection device, comprising:

[0027] Processor and transceiver interface;

[0028] The transceiver interface is used to acquire videos, including those of the user.

[0029] The processor is configured to determine a first motion index of the user during the recording of the video including the user, and based on the first motion index, determine at least one first video segment included in the video, in which the user's motion amplitude is less than a first threshold corresponding to the first motion index, and then determine the user's heart rate based on the first video segment.

[0030] In one optional design, the first motion index includes at least one of the following: the displacement of the user's head, the angle change of the user's head, the displacement of m feature points of the user, the displacement of n grids, and the distance change between the grids;

[0031] Where m and n are positive integers, and the grid is the grid obtained by dividing the user's contour.

[0032] In one optional design, the processor is specifically configured to determine the heart rate spectrum curve corresponding to the first video segment based on the images included in the first video segment;

[0033] If the first video segment includes at least two elements, determine the weight of the first video segment;

[0034] The user's heart rate is determined based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment;

[0035] If the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

[0036] In one optional design, the processor is specifically configured to determine the weight of the first video segment based on the duration of the first video segment, or to determine the weight of the first video segment based on the part of the user included in the first video segment.

[0037] In one optional design, the processor is specifically configured to: determine a first curve of the first video segment based on the photoplethysmography (PPG) signal of the same region in each frame of the first video segment, wherein the horizontal axis of the first curve is time and the vertical axis is the PPG signal, and the same region in each frame includes the user; obtain a second curve after bandpass filtering by performing bandpass filtering on the first curve; and determine the heart rate spectrum curve corresponding to the first video segment by performing fast Fourier transform on the second curve.

[0038] Thirdly, embodiments of this application disclose a terminal device, including:

[0039] At least one processor and memory,

[0040] The memory is used to store program instructions;

[0041] The processor is configured to call and execute program instructions stored in the memory to cause the terminal device to perform the heart rate detection method described in the first aspect.

[0042] Fourthly, embodiments of this application disclose a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the heart rate detection method as described in the first aspect.

[0043] Fifthly, embodiments of this application disclose a computer program product containing instructions that, when the computer program product is run on an electronic device, enable the electronic device to implement the heart rate detection method described in the first aspect.

[0044] In this embodiment, since the user's movement amplitude is less than the first threshold corresponding to the first motion index within the first video segment, the user remains stationary or exhibits minimal movement within the first video segment. Consequently, the same area in each frame of the first video segment often includes the same part of the user's body. In this case, the accuracy of determining the user's heart rate through the first video segment is relatively high. In other words, the solution provided by this embodiment can improve the accuracy of heart rate detection. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of a heart rate spectrum curve;

[0046] Figure 2 This is a schematic diagram of the structure of a terminal device disclosed in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating the workflow of a heart rate detection method disclosed in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram illustrating the workflow of a heart rate detection method disclosed in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the heart rate spectrum curve disclosed in the embodiments of this application;

[0050] Figure 6 This is a schematic diagram of the structure of a heart rate detection device disclosed in an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of the structure of a terminal device disclosed in an embodiment of this application. Detailed Implementation

[0052] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0053] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only. In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0055] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0056] Heart rate serves as an important indicator for assessing human health. As people become increasingly concerned about their own and their families' health, heart rate monitoring is gaining more and more attention. To meet this demand, some devices, such as mobile phones or watches, now offer heart rate monitoring functionality. Users can monitor their heart rate using these devices.

[0057] Currently, terminal devices typically employ the following techniques when detecting a user's heart rate: First, a video of the user's face is captured over a period of time. During the capture process, the user is usually required to remain still so that the same area of ​​the user's face is displayed in the multiple frames of the video. Then, based on the effect presented at the same location in the multiple frames, the photoplethysmograph (PPG) signal of the skin at that location at different times is determined. Next, the PPG signal is processed to determine the user's heart rate spectrum curve. The horizontal axis of the heart rate frequency curve is usually time, and the vertical axis of the heart rate spectrum curve is usually the PPG signal, which can be used to represent the PPG signal corresponding to each time point. Finally, the user's heart rate is determined using this heart rate spectrum curve.

[0058] However, existing technologies for detecting heart rate typically use videos that are at least 10 seconds long. Users are unlikely to remain completely still for 10 seconds; in other words, users often move during the process of capturing facial video.

[0059] Because users typically move during video capture, the same location in multiple frames of the video may represent different parts of the user's face. Therefore, the PPG signals determined at different times from these multiple frames may correspond to different areas of the user's face, resulting in lower accuracy of the user's heart rate determined by existing technologies.

[0060] In one example of detecting a user's heart rate using existing technology, the heart rate frequency curve determined from a video including the user's face can be as follows: Figure 1 As shown. In Figure 1 The image includes a vertical line segment. The left side of the line segment represents the heart rate spectrum curve determined from multiple frames of images of the user at rest; this curve can be called the first curve. The right side of the line segment represents the heart rate spectrum curve determined from multiple frames of images of the user in motion; this curve can be called the second curve. The horizontal axis of both curves represents time, and the vertical axis represents the intensity of the PPG signal.

[0061] The first curve is relatively stable, and the PPG signal under this curve corresponds to the skin of the same area on the user's face. However, at the time corresponding to the second curve, due to user movement, the PPG signal at different times often corresponds to different areas of the user's face, making the second curve less stable.

[0062] In this case, if through Figure 1 The curve shown determines the user's heart rate, but because the entire curve is determined by the PPG signal of different areas of the user's face, the accuracy of the detected heart rate is relatively low.

[0063] To address the issue of low accuracy in existing heart rate detection technologies, this application provides a heart rate detection method and apparatus.

[0064] The heart rate detection method provided in this application can be applied to a terminal device, which can be of various types. The terminal device can acquire video including the user, or it can receive video including the user acquired by other imaging devices, so as to determine the user's heart rate through processing the video.

[0065] In some embodiments, the terminal device may be a mobile phone, tablet computer, desktop computer, laptop computer, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, etc. This application does not impose any special restrictions on the specific form of the terminal device.

[0066] Taking a smartphone as an example, the structural diagram of the terminal device can be as follows: Figure 2 As shown. See also Figure 2 The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a radio frequency module 150, a communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a screen 301, and a subscriber identification module (SIM) card interface 195, etc.

[0067] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone. In other embodiments of this application, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0068] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0069] The controller can serve as the nerve center and command center of the mobile phone. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0070] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0071] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0072] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thus realizing the touch function of the mobile phone.

[0073] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0074] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0075] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the communication module 160. For example, the processor 110 communicates with the Bluetooth module in the communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0076] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the screen 301 and camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and camera 193 communicate via the CSI interface to enable the phone's camera function. The processor 110 and screen 301 communicate via the DSI interface to enable the phone's display function.

[0077] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a screen 301, a communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0078] USB port 130 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge a mobile phone, or to transfer data between a mobile phone and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0079] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the mobile phone. In other embodiments of this application, the mobile phone may also adopt different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0080] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the mobile phone. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0081] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, screen 301, camera 193, and communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0082] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, radio frequency module 150, communication module 160, modem processor, and baseband processor.

[0083] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0084] The radio frequency (RF) module 150 can provide solutions for wireless communication applications in mobile phones, including 2G / 3G / 4G / 5G. The RF module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The RF module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The RF module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the RF module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the RF module 150 and at least some modules of the processor 110 may be housed in the same device.

[0085] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through screen 301. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the radio frequency module 150 or other functional modules.

[0086] The communication module 160 can provide solutions for wireless communication applications in mobile phones, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The communication module 160 can be one or more devices integrating at least one communication processing module. The communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0087] In some embodiments, antenna 1 of the mobile phone is coupled to radio frequency module 150, and antenna 2 is coupled to communication module 160, enabling the mobile phone to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0088] The mobile phone implements its display function through a GPU, screen 301, and application processor. The GPU is a microprocessor for image processing, connected to the screen 301 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. In this embodiment, the screen 301 may include a display and a touch device. The display outputs content to the user, and the touch device receives touch events input by the user on the screen 301.

[0089] In a mobile phone, the sensor module 180 may include one or more of the following: gyroscope, accelerometer, pressure sensor, barometric pressure sensor, magnetic sensor (e.g., Hall sensor), proximity sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, pyroelectric infrared sensor, ambient light sensor, or bone conduction sensor. This application embodiment does not impose any limitations on these.

[0090] Mobile phones can achieve shooting functions through ISP, camera 193, video codec, GPU, flexible screen 301 and application processor.

[0091] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0092] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the mobile phone may include one or N cameras 193, where N is a positive integer greater than 1.

[0093] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when a mobile phone is selecting a frequency, the DSP performs Fourier transforms on the frequency energy.

[0094] Video codecs are used to compress or decompress digital video. A mobile phone can support one or more video codecs. This allows the phone to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0095] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in mobile phones, such as image recognition, facial recognition, speech recognition, and text understanding.

[0096] The external storage interface 120 can be used to connect an external storage card, such as a MicroSD card, to expand the phone's storage capacity. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0097] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various mobile phone functions and data processing by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during mobile phone use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0098] Mobile phones can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. These functions include network standard determination and recording.

[0099] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0100] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Mobile phones can use the speaker 170A to listen to music or make hands-free calls.

[0101] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When answering a phone call or voice message, the receiver 170B can be brought close to the user's ear to hear the voice.

[0102] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. A mobile phone can have at least one microphone 170C. In some embodiments, a mobile phone can have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, a mobile phone can have three, four, or more microphones 170C, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0103] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0104] Button 190 includes the power button, volume buttons, etc. Button 190 can be a mechanical button or a touch button. The mobile phone can receive button input and generate key signal inputs related to the phone's user settings and function control.

[0105] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the flexible screen 301. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0106] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0107] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile phone. The mobile phone can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external storage cards. The mobile phone interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the mobile phone uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone and cannot be separated from the mobile phone.

[0108] In addition, an operating system runs on top of these components. Examples include Apple's iOS operating system, Google's Android open-source operating system, and Microsoft's Windows operating system. Applications can be installed and run on this operating system.

[0109] To clarify the solution provided in this application, the solution provided in this application will be described and explained below with reference to the accompanying drawings and various embodiments.

[0110] See Figure 3 The flowchart shown illustrates the heart rate detection method provided in this application embodiment, which includes the following steps:

[0111] Step S11: Determine the user's first motion index during the process of shooting a video including the user.

[0112] In this embodiment, the first motion indicator is used to indicate whether the user has moved. The video may include the user, and the video may include the user's head or other parts of the user's body. In this case, the first motion indicator during the video recording process can determine whether the user has moved during the recording process.

[0113] The first motion index may include multiple parameters. In one feasible implementation, the first motion index includes at least one of the following: the displacement of the user's head, the angle change of the user's head, the displacement of m feature points of the user, the displacement of n grids, and the distance change between the grids. Wherein, m and n are positive integers, and the grid is a grid obtained by dividing the user's contour.

[0114] If the video captures a user's head, the movement of the user can be determined based on the displacement and / or the change in the angle of the user's head. For example, if the user looks up, their head will tilt upwards and move upwards; therefore, the movement of the user can be determined by the displacement and / or the change in the angle of their head.

[0115] Generally, the greater the displacement of the user's head, and / or the greater the change in the angle of the user's head, the greater the range of motion of the user.

[0116] In this embodiment, the displacement of the user's head and / or the range of change in the angle of the user's head can be acquired using 6 degrees of freedom (DOF) technology. In this scheme, the user can wear a virtual reality (VR) device, which can acquire the displacement of the user's head and / or the range of change in the angle of the user's head, and transmit the acquired displacement of the user's head and / or the range of change in the angle of the user's head to a terminal device executing the heart rate detection method provided in this embodiment.

[0117] Of course, in the embodiments of this application, the displacement of the user's head and / or the range of change in the angle of the user's head can also be determined by other means, and the embodiments of this application do not limit this.

[0118] Furthermore, in this embodiment, the displacement of feature points can also be used to determine whether the user has moved. Here, a feature point typically refers to a location on the user's body. For example, if the video captures the user's face, a feature point can be a specific location on the user's face; specifically, the feature point may include the tip of the nose, the corner of the mouth, and the corner of the eye, etc.

[0119] When a user moves, their feature points will shift. For example, when a user smiles, the corners of their mouth tend to move upwards. Generally, the greater the shift, the greater the range of motion.

[0120] In this embodiment, the displacement of the feature points can be determined based on the coordinates of feature points in each frame of the video and the scaling ratio between the image and the actual user. Of course, the displacement of the feature points can also be determined in other ways, and this embodiment does not limit this method.

[0121] m is a positive integer. In a feasible design, the value of m can range from 10 to 100,000. Of course, m can also be a smaller or larger value. The specific value of m can be determined according to the accuracy requirements of heart rate detection. Generally, the higher the accuracy requirement of heart rate detection, the larger the value of m.

[0122] Additionally, in this embodiment, the user's outline can be divided into multiple grids, each corresponding to a specific location on the user's body. When dividing the outline, firstly, a planar model of the user's outline can be determined based on the first few frames of the video, and then this planar model can be divided into multiple grids. Alternatively, when dividing the outline, firstly, a three-dimensional model of the user can be determined based on the first few frames of the video, and then this three-dimensional model can be divided into multiple grids.

[0123] After the grid is divided, the displacement of the grid can be determined based on the coordinates of each grid in each frame of the video and the scaling ratio between the image and the actual user. The magnitude of the distance change between the grids can also be determined.

[0124] Since the grid is obtained by dividing the user's contours, each grid often corresponds to a specific location on the user's body, and the displacement of this grid reflects the user's movement. For example, if the user smiles, the grid corresponding to the corner of the user's mouth will often shift upwards. Furthermore, generally, the greater the displacement of the grid, the greater the range of the user's movement.

[0125] Furthermore, n is a positive integer. In a feasible design, the value of n can range from 10 to 100,000. Of course, n can also be a smaller or larger value. The specific value of n can be determined according to the accuracy requirements of heart rate detection. Generally, the higher the accuracy requirement for heart rate detection, the larger the value of n. Also, generally, the higher the accuracy of heart rate detection, the smaller the area of ​​the grid, that is, the more grids are obtained after dividing the user contour.

[0126] Since each grid corresponds to a specific location on the user's body, the distance between different grids usually changes during the user's movement. For example, when a user smiles, the distance between the two grids corresponding to the corners of the mouth changes. Therefore, in this embodiment, the first motion index may further include the magnitude of the distance change between the grids.

[0127] Of course, the first motion index may also include other indicators that can reflect the user's range of motion, and this application embodiment does not limit this.

[0128] Step S12: Based on the first motion index, determine at least one first video segment included in the video, in which the user's motion amplitude is less than the first threshold corresponding to the first motion index.

[0129] In this embodiment of the application, within the first video segment, the user's movement amplitude is less than the first threshold corresponding to the first movement index. Therefore, it can be considered that within the first video segment, the user is stationary or the user's movement amplitude is small.

[0130] In this step, the first threshold is different for different first motion indicators. To clarify the first threshold corresponding to each first motion indicator, examples of the first threshold corresponding to each first motion indicator are provided below.

[0131] (1) In a feasible design, if the first motion index includes the displacement of the user's head, the value range of the first threshold corresponding to this index can be 0.1 mm to 2 cm. The specific value of the first threshold corresponding to this index can be determined according to the accuracy requirements of heart rate detection. Generally, the higher the accuracy requirements of heart rate detection, the smaller the first threshold corresponding to the displacement of the user's head.

[0132] In this case, if the displacement of the user's head is less than the first threshold within a certain video segment, then the video segment can be identified as the first video segment.

[0133] (2) If the first motion index includes the angle change of the user's head, the value range of the first threshold corresponding to this index can be 0.1 degrees to 5 degrees. The specific value of the first threshold corresponding to this index can be determined according to the accuracy requirements of heart rate detection. Generally, the higher the accuracy requirements of heart rate detection, the smaller the first threshold corresponding to the angle change of the user's head.

[0134] In this case, if the angle of the user's head changes less than the first threshold within a certain video segment, then the video segment can be identified as the first video segment.

[0135] (3) If the first motion index includes the displacement of m feature points of the user, the value range of the first threshold corresponding to this index can be 0.1 mm to 2 cm. The specific value of the first threshold corresponding to this index can be determined according to the accuracy requirements of heart rate detection. Generally, the higher the accuracy requirements of heart rate detection, the smaller the first threshold corresponding to the displacement of m feature points of the user.

[0136] In this case, if the displacement of the m feature points is always less than the first threshold within a certain video segment, then the video segment can be determined as the first video segment.

[0137] (4) The first motion index may include the displacement of the n grids. The first threshold corresponding to the displacement of the n grids may be determined based on the area of ​​the outline of the user used to divide the grids and the number of grids. The smaller the area or the more grids there are, the smaller the first threshold corresponding to the displacement of the n grids tends to be.

[0138] Alternatively, the higher the accuracy requirement for heart rate detection, the smaller the first threshold corresponding to the displacement of the n grids.

[0139] In this case, if the displacement of the n grids is always less than the first threshold within a certain video segment, then the video segment can be determined as the first video segment.

[0140] (5) The first motion index may include the distance change range between the grids. The first threshold corresponding to the distance change range between the grids may also be determined based on the area of ​​the outline of the user used to divide the grids and the number of grids. The smaller the area or the more grids there are, the smaller the first threshold corresponding to the distance change range between the grids is usually.

[0141] Alternatively, the higher the accuracy requirement for heart rate detection, the smaller the first threshold corresponding to the indicator of the change in distance between the grids.

[0142] In this case, if the distance between the grids within a certain video segment is consistently less than the first threshold, then the video segment can be identified as the first video segment.

[0143] Of course, the first threshold corresponding to the first motion index can also be other values, and this application embodiment does not limit this.

[0144] Step S13: Determine the user's heart rate based on the first video segment.

[0145] In this embodiment, since the user's movement amplitude is less than the first threshold corresponding to the first motion index within the first video segment, the user remains stationary or exhibits minimal movement within the first video segment. Consequently, the same area in each frame of the first video segment often includes the same part of the user's body. In this case, the accuracy of determining the user's heart rate through the first video segment is relatively high. In other words, the solution provided by this embodiment can improve the accuracy of heart rate detection.

[0146] In this embodiment of the application, after determining the first video segment, the user's heart rate is determined based on the first video segment. See also Figure 4 The workflow diagram shown illustrates that this operation can be achieved through the following steps:

[0147] Step S21: Determine the heart rate spectrum curve corresponding to the first video segment based on the images included in the first video segment.

[0148] In this context, the horizontal axis of the heart rate spectrum curve corresponding to the first video segment is usually time, while the vertical axis represents the PPG signal. In other words, the heart rate spectrum curve corresponding to the first video segment can indicate the PPG signal at each moment and the change pattern of the PPG signal over time.

[0149] Step S22: If the first video segment includes at least two segments, determine the weight of the first video segment.

[0150] During the recording of the first video, the user may remain still or move only slightly for one or more periods of time. In this case, the first video may include at least two first video segments.

[0151] For example, see the reference. Figure 5 The diagram illustrates a heart rate spectrum curve, which includes three boxes, each containing a segment of the curve. In other words, these three boxes divide the heart rate spectrum curve into three segments. When the video segment corresponding to the heart rate spectrum curve in the middle box was filmed, the user's movement was significant, causing the heart rate spectrum curve within that box to be unstable and fluctuate considerably. In this case, the video segment corresponding to the middle box is generally not considered the first video segment, and consequently, the heart rate spectrum curve within the middle box is not the heart rate spectrum curve corresponding to the first video segment.

[0152] The heart rate spectrum curves in the leftmost and rightmost boxes are relatively stable. When capturing the video segments corresponding to these heart rate spectrum curves, the user remains still or experiences minimal movement. In this case, both video segments corresponding to these heart rate spectrum curves can be used as the first video segment. That is to say, in Figure 5 In this corresponding example, two first video segments may be included.

[0153] If the first video segment includes at least two segments, the user's heart rate needs to be determined by the heart rate spectrum curves corresponding to the at least two first video segments. Therefore, it is necessary to determine the weight of each first video segment.

[0154] The weight of the first video segment can be determined in a variety of ways.

[0155] In one feasible implementation, determining the weight of the first video segment includes the following steps:

[0156] The weight of the first video segment is determined based on its duration.

[0157] In this implementation, the longer the first video segment, the higher its weight. In one feasible design, the weight of the first video segment can be determined by the following formula:

[0158] The weight of the first video segment = the duration of the first video segment / the sum of the durations of all first video segments.

[0159] For example, if the video includes two first video segments, the sum of the durations of the two first video segments is 10 seconds, and the duration of one of the first video segments is 4 seconds, then the weight of that first video segment is 0.4.

[0160] Alternatively, in another feasible implementation, determining the weight of the first video segment includes the following steps:

[0161] The weight of the first video segment is determined based on the user's body parts included in the first video segment.

[0162] If the user moves during video recording, different first video clips may capture different parts of the user. In this implementation, a score can be set for each part of the user's face. For example, if the video including the user includes the user's face, scores can be pre-set for each part of the user's face (e.g., chin, corners of the mouth, and eyes). When determining the weight of a particular first video clip, the score corresponding to the user portion contained in each frame of that first video clip is determined, and this score is used as the score of that first video clip. In this case, the weight of the first video clip can be determined using the following formula:

[0163] The weight of the first video segment = the score of the first video segment / the sum of the scores of all first video segments.

[0164] Of course, the weight of the first video segment can also be determined by other means, and this application embodiment does not limit this.

[0165] Step S23: Determine the user's heart rate based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment.

[0166] In this step, the heart rate corresponding to each first video segment can be determined based on the heart rate frequency curve corresponding to each first video segment. Then, the user's heart rate can be determined based on the weight of each first video segment and the heart rate corresponding to each first video segment.

[0167] Specifically, when determining the heart rate corresponding to each first video segment, the frequency of the heart rate spectrum curve can be determined based on the heart rate spectrum curve of the first video segment, and then the heart rate corresponding to the first video segment can be determined based on the frequency of the heart rate spectrum curve.

[0168] For example, if the frequency of a certain first video segment is 1Hz, then the heart rate corresponding to the first video segment is 60 beats per minute.

[0169] Furthermore, since a user's heart rate typically falls within a certain range, the frequency of the heart rate spectrum curve should generally be between 0.83 and 2.67 Hz. Therefore, in the solution provided in this application embodiment, if the frequency of a certain first video segment is greater than 2.67 Hz or less than 0.83 Hz, it can be determined that the heart rate corresponding to that first video segment is inaccurate. In this case, the user's heart rate will no longer be determined based on the heart rate corresponding to that first video segment, thereby further improving the accuracy of heart rate detection.

[0170] In addition, when determining the user's heart rate based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment, the following formula can be used:

[0171]

[0172] Where s is the total number of the first video segments, and i and s are both positive integers.

[0173] For example, the video includes two first video segments. The heart rate corresponding to the first first video segment is 60 beats / minute, and the weight of this first video segment is 0.4. The heart rate corresponding to the second first video segment is 80 beats / minute, and the weight of this first video segment is 0.6. Based on these two first video segments, the user's heart rate can be determined as 60*0.4 + 80*0.6 = 72 (beats / minute).

[0174] In addition, if the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

[0175] In the above scheme, the operation of publicly determining the heart rate spectrum curve corresponding to the first video segment can be achieved through the following steps:

[0176] The first step is to determine the first curve of the first video segment by using photoplethysmography (PPG) to record the PPG signal in the same area of ​​each frame image in the first video segment. The horizontal axis of the first curve is time, and the vertical axis is the PPG signal. The same area of ​​each frame image includes the user.

[0177] Since the user was stationary or moved only slightly during the recording of the first video segment, the same area in each frame often contains the same part of the user's body. In this case, the first curve can be determined based on the PPG signal of the same area in each frame of the first video segment.

[0178] The PPG signal can be determined by the green color component of the same region in each frame of the image. Furthermore, when determining the PPG signal, the user portion and non-user portions of the image can also be identified. Based on the non-user portions, the PPG signal of the user's environment is determined, and environmental interference from the user's environment's PPG signal on the user portion is eliminated. Then, the first curve is determined based on the image after eliminating environmental interference.

[0179] The second step is to obtain a second curve after bandpass filtering by performing bandpass filtering on the first curve.

[0180] The third step involves performing a Fast Fourier Transform (FFT) on the second curve to determine the heart rate spectrum curve corresponding to the first video segment. The second curve after the FFT is the heart rate spectrum curve corresponding to the first video segment.

[0181] By following the steps described above, the heart rate spectrum curve corresponding to the first video segment can be determined, so that the user's heart rate can be determined using the heart rate spectrum curve corresponding to the first video segment.

[0182] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0183] As an implementation of the above embodiments, this application discloses a heart rate detection device. See also... Figure 6 The schematic diagram shown indicates that the heart rate detection device includes a processor 1110 and a transceiver interface 1120.

[0184] The transceiver interface is used to acquire videos, including those of the user.

[0185] The processor is configured to determine a first motion index of the user during the recording of the video including the user, and based on the first motion index, determine at least one first video segment included in the video, in which the user's motion amplitude is less than a first threshold corresponding to the first motion index, and then determine the user's heart rate based on the first video segment.

[0186] In the solution provided in this application embodiment, the first motion index includes at least one of the following: the displacement of the user's head, the angle change of the user's head, the displacement of m feature points of the user, the displacement of n grids, and the distance change between the grids. Wherein, m and n are positive integers, and the grid is a grid obtained by dividing the user's contour.

[0187] In the solution provided in this application embodiment, the transceiver interface can be connected to an imaging device, which is used to capture video including the user, and the transceiver interface can acquire the video captured by the imaging device. Then, the processor determines the user's heart rate through the video.

[0188] The processor can determine the user's first motion index in multiple ways. Specifically, if the first motion index includes the displacement of the user's head and / or the range of change in the user's head angle, and the displacement of the user's head and / or the range of change in the user's head angle are collected by a VR device, the transceiver interface can also be connected to the VR device and transmit the displacement of the user's head and / or the range of change in the user's head angle collected by the VR device to the processor, so that the processor can determine the first motion index.

[0189] In addition, if the first motion index includes other motion indexes, and the other motion indexes need to be determined by information collected by other sensors, the transceiver interface can also be connected to other sensors and receive the information collected by the other sensors, and then transmit the information collected by the other sensors to the processor so that the processor can determine the other motion indexes based on the information.

[0190] Since the user's movement amplitude in the first video segment is less than the first threshold corresponding to the first motion index, the user remains stationary or has a small movement amplitude within the first video segment. Consequently, the same area in each frame of the first video segment often includes the same part of the user's body. In this case, the accuracy of determining the user's heart rate through the first video segment is relatively high. In other words, the solution provided in this application embodiment can improve the accuracy of heart rate detection.

[0191] In one feasible design, the processor is specifically configured to determine the heart rate spectrum curve corresponding to the first video segment based on the images included in the first video segment;

[0192] If the first video segment includes at least two elements, determine the weight of the first video segment;

[0193] The user's heart rate is determined based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment;

[0194] If the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

[0195] Specifically, the processor is used to determine the weight of the first video segment based on its duration, or to determine the weight of the first video segment based on the user's part included in the first video segment.

[0196] Furthermore, in a feasible design, the processor is specifically used to determine a first curve of the first video segment based on the photoplethysmography (PPG) signal of the same region in each frame of the first video segment, wherein the horizontal axis of the first curve is time and the vertical axis is the PPG signal, and the same region in each frame of the first video segment includes the user; and to obtain a second curve after bandpass filtering by performing bandpass filtering on the first curve, and then to determine the heart rate spectrum curve corresponding to the first video segment by performing fast Fourier transform on the second curve.

[0197] Correspondingly, in accordance with the methods described above, this application also discloses a terminal device. See [link to related document]. Figure 7 The schematic diagram shown indicates that the terminal device includes:

[0198] At least one processor 1101 and memory,

[0199] The memory is used to store program instructions;

[0200] The processor is configured to call and execute program instructions stored in the memory, so that the terminal device performs... Figure 3 and Figure 4 All or part of the steps in the corresponding embodiments.

[0201] Furthermore, the terminal device may also include a transceiver 1102 and a bus 1103, and the memory includes a random access memory 1104 and a read-only memory 1105.

[0202] The processor is coupled to the transceiver, random access memory (RAM), and read-only memory (ROM) via a bus. When the terminal device needs to run, it is booted via a basic input / output system (BIS) embedded in the ROM or a bootloader within the embedded system, entering normal operating mode. Once in normal operating mode, the application program and operating system run in the RAM, enabling the terminal device to perform its functions. Figure 3 and Figure 4 All or part of the steps in the corresponding embodiments.

[0203] The apparatus of this invention can correspond to the above. Figure 3 and Figure 4 The corresponding embodiment includes a heart rate detection device, and the processor in that device can implement... Figure 3 and Figure 4 For the sake of brevity, the functions of the heart rate detection device in the corresponding embodiments, or the various steps and methods implemented, will not be described in detail here.

[0204] In the embodiments of this application, the terminal device can take various forms. For example, the terminal device may include a mobile phone. In this case, the mobile phone's camera, as an imaging device, can capture video including the user. The mobile phone's processor can then detect the heart rate based on the video including the user, using the method provided in the embodiments of this application.

[0205] Alternatively, the terminal device may include a smart mirror. In this case, while the user is looking in the mirror, the imaging device of the smart mirror can capture a video including the user, and the processor of the smart mirror can detect the heart rate based on the video including the user, using the method provided in the embodiments of this application. This allows the system to output the user's heart rate detection result while the user is looking in the mirror, thus meeting the user's heart rate detection needs.

[0206] Additionally, the terminal device may include a television set. In this case, while the user is watching television or facing the television set, the television's imaging device can capture video including the user, and the television's processor can detect the heart rate based on the video including the user, using the method provided in this application embodiment. This allows the system to output the user's heart rate detection result while the user is watching television, thus meeting the user's heart rate detection needs.

[0207] Of course, the terminal device may also take other forms, and this application embodiment does not limit this.

[0208] In a specific implementation, embodiments of this application also provide a computer-readable storage medium, which includes instructions. Wherein, a computer-readable medium disposed in any device, when executed on a computer, can implement instructions including... Figure 3 and Figure 4 All or part of the steps in the corresponding embodiments. The storage medium of the computer-readable medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0209] Additionally, another embodiment of this application discloses a computer program product containing instructions that, when executed on an electronic device, enable the electronic device to implement, including... Figure 3 and Figure 4 All or part of the steps in the corresponding embodiments.

[0210] The various illustrative logic units and circuits described in the embodiments of this application can be implemented or operate the described functions using a general-purpose processor, digital information processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital information processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital information processor core, or any other similar configuration.

[0211] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be disposed in an ASIC, which can be disposed in the UE. Optionally, the processor and storage medium can also be disposed in different components within the UE.

[0212] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0213] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0214] The same or similar parts between the various embodiments in this specification can be referred to interchangeably. Each embodiment focuses on the differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to in the description of the method embodiments section.

[0215] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0216] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the embodiments of the road constraint determination device disclosed in this application are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0217] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A heart rate detection method, characterized in that, include: During the recording of a video including the user, a first motion index of the user is determined; the first motion index includes at least one of the following: the displacement of the user's head, the angle change of the user's head, the displacement of m feature points of the user, the displacement of n grids, and the distance change between the grids, where m and n are positive integers, the grid is a grid obtained by dividing the user's contour, the grid is a grid obtained by dividing a planar model of the user's contour, or the grid is a grid obtained by dividing a three-dimensional model of the user's contour, the planar model or the three-dimensional model is used to determine based on the video; Based on the first motion index, at least one first video segment is determined to be included in the video, in which the user's motion amplitude is less than a first threshold corresponding to the first motion index; wherein, if the first motion index includes the displacement of the user's head, the value range of the first threshold corresponding to the first motion index is 0.1 mm to 2 cm; if the first motion index includes the angle change of the user's head, the first threshold corresponding to the first motion index is 0.1 degrees to 5 degrees; if the first motion index includes the displacement of m feature points of the user, the first threshold corresponding to the first motion index is 0.1 mm to 2 cm; if the first motion index includes the displacement of n grids, the first threshold corresponding to the displacement of the n grids is determined based on the area of ​​the user's contour used to divide the grids and the number of grids; if the first motion index includes the distance change between the grids, the first threshold corresponding to the distance change between the grids is determined based on the area of ​​the user's contour used to divide the grids and the number of grids. The user's heart rate is determined based on the first video segment.

2. The method according to claim 1, characterized in that, Determining the user's heart rate based on the first video segment includes: Based on the images included in the first video segment, determine the heart rate spectrum curve corresponding to the first video segment; If the first video segment includes at least two elements, determine the weight of the first video segment; The user's heart rate is determined based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment; If the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

3. The method according to claim 2, characterized in that, Determining the weight of the first video segment includes: The weight of the first video segment is determined based on its duration. Alternatively, the weight of the first video segment can be determined based on the user's body parts included in the first video segment.

4. The method according to claim 2, characterized in that, Determining the heart rate spectrum curve corresponding to the first video segment includes: Based on the photoplethysmography (PPG) signal of the same region in each frame of the first video segment, a first curve of the first video segment is determined. The horizontal axis of the first curve is time, and the vertical axis is the PPG signal. The same region in each frame includes the user. By applying a bandpass filter to the first curve, a second curve that has undergone bandpass filtering is obtained. The heart rate spectrum curve corresponding to the first video segment is determined by performing a fast Fourier transform on the second curve.

5. A heart rate detection device, characterized in that, include: Processor and transceiver interface; The transceiver interface is used to acquire videos, including those of the user. The processor is configured to: determine a first motion index of the user during the recording of the video including the user; determine at least one first video segment included in the video based on the first motion index, wherein the user's motion amplitude in the first video segment is less than a first threshold corresponding to the first motion index; and determine the user's heart rate based on the first video segment. The first motion index includes at least one of the following: displacement of the user's head, change in the angle of the user's head, displacement of m feature points of the user, displacement of n grids, and change in distance between the grids, wherein m and n are positive integers; the grid is a grid obtained by dividing the user's contour; the grid is a grid obtained by dividing a planar model of the user's contour; or the grid is a grid obtained by dividing the contour of a three-dimensional model of the user. A body model is used to determine the motion model based on the video; wherein, if the first motion indicator includes the displacement of the user's head, the first threshold corresponding to the first motion indicator ranges from 0.1 mm to 2 cm; if the first motion indicator includes the angle change of the user's head, the first threshold corresponding to the first motion indicator is from 0.1 degrees to 5 degrees; if the first motion indicator includes the displacement of m feature points of the user, the first threshold corresponding to the first motion indicator is from 0.1 mm to 2 cm; if the first motion indicator includes the displacement of n grids, the first threshold corresponding to the displacement of the n grids is determined based on the area of ​​the user's contour used to divide the grids and the number of grids; if the first motion indicator includes the distance change between the grids, the first threshold corresponding to the distance change between the grids is determined based on the area of ​​the user's contour used to divide the grids and the number of grids.

6. The apparatus according to claim 5, characterized in that, The processor is specifically used to determine the heart rate spectrum curve corresponding to the first video segment based on the images included in the first video segment. If the first video segment includes at least two elements, determine the weight of the first video segment; The user's heart rate is determined based on the weight of the first video segment and the heart rate spectrum curve corresponding to the first video segment; If the first video segment is a single video clip, the user's heart rate is determined based on the heart rate spectrum curve corresponding to the first video segment.

7. The apparatus according to claim 6, characterized in that, The processor is specifically configured to determine the weight of the first video segment based on its duration, or to determine the weight of the first video segment based on the user's part included in the first video segment.

8. The apparatus according to claim 6, characterized in that, The processor is specifically configured to: determine a first curve of the first video segment based on the photoplethysmography (PPG) signal of the same region in each frame of the first video segment, wherein the horizontal axis of the first curve is time and the vertical axis is the PPG signal, and the same region in each frame of the first video segment includes the user; obtain a second curve after bandpass filtering by performing bandpass filtering on the first curve; and determine the heart rate spectrum curve corresponding to the first video segment by performing fast Fourier transform on the second curve.

9. A terminal device, characterized in that, include: At least one processor and memory, The memory is used to store program instructions; The processor is configured to call and execute program instructions stored in the memory to cause the terminal device to perform the heart rate detection method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the heart rate detection method as described in any one of claims 1-4.

Images