Non-contact user heart rate monitoring method and device, electronic equipment and storage medium

Abstract

The invention discloses a non-contact type user heart rate monitoring method and device, electronic equipment and a storage medium, which are used for solving the problem of poor heart rate monitoring accuracy existing in a related non-contact type heart rate monitoring technology. The method comprises the following steps: acquiring a plurality of face images of a user; determining a target region of interest in the plurality of face images based on a preset target prompt word and a pre-trained multi-modal language model; wherein the target cue word is used for enabling the pre-trained multi-modal language model to screen an area in the face image, so as to output an area of interest in the face image; and determining heart rate data of the user based on the target region of interest in the plurality of face images.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a non-contact user heart rate monitoring method, device, electronic device and storage medium. Background Art

[0002] Traditional heart rate monitoring requires people to wear corresponding sensors, such as electrodes for measuring electrocardiograms, which can cause discomfort if worn for a long time. Video-based heart rate monitoring, such as remote photoplethysmography (rPPG), can use ordinary cameras to record changes in skin surface color caused by heartbeat activity, and thereby achieve contactless measurement of heart rate. Therefore, face video contactless heart rate monitoring based on rPPG technology has received widespread attention. Among them, selecting a suitable skin area as the region of interest (ROI) is a crucial step in the entire algorithm process, because this step directly affects the quality of subsequent signal extraction and the accuracy of heart rate estimation.

[0003] In related non-contact heart rate monitoring solutions, such as those using rPPG technology, regions of interest are usually selected from video frames according to preset rules. For example, traditional methods usually rely on color thresholds or morphological operations to identify and segment skin areas, while deep learning methods can automatically learn skin features by training convolutional neural networks using attention mechanisms to determine regions of interest.

[0004] However, when used in real application scenarios, the skin screening strategy based on preset rules fails due to the complexity of the actual environment, resulting in inaccurate determination of the area of ​​interest and poor accuracy of heart rate monitoring.

[0005] In other words, the relevant non-contact heart rate monitoring technology has the problem of poor accuracy of heart rate monitoring. Summary of the invention

[0006] The embodiments of the present application provide a non-contact user heart rate monitoring method, device, electronic device and storage medium, which can solve the problem of poor accuracy of heart rate monitoring in related non-contact heart rate monitoring technologies.

[0007] In a first aspect, an embodiment of the present application provides a contactless user heart rate monitoring method, the method comprising: acquiring multiple facial images of a user; determining a target region of interest in the multiple facial images based on a preset target prompt word and a pre-trained multimodal language model; wherein the target prompt word is used to enable the pre-trained multimodal language model to filter the region in the facial image to output the region of interest in the facial image; and determining the user's heart rate data based on the target region of interest in the multiple facial images.

[0008] In a second aspect, an embodiment of the present application provides a contactless user heart rate monitoring device, the device comprising: an acquisition module, used to acquire multiple facial images of a user; a first determination module, used to determine a target region of interest in the multiple facial images based on a preset target prompt word and a pre-trained multimodal language model; wherein the target prompt word is used to enable the pre-trained multimodal language model to filter the region in the facial image to output the region of interest in the facial image; a second determination module, used to determine the user's heart rate data based on the target region of interest in the multiple facial images.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the non-contact user heart rate monitoring method as described in the first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium is used to store computer-executable instructions, wherein the computer-executable instructions enable a computer to execute the non-contact user heart rate monitoring method as described in the first aspect.

[0011] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the non-contact user heart rate monitoring method as described in the first aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the non-contact user heart rate monitoring method as described in the first aspect.

[0013] In an embodiment of the present application, a plurality of facial images of a user are first obtained; then, based on a preset target prompt word and a pre-trained multimodal language model, a target region of interest in the plurality of facial images is determined; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the facial image to output the region of interest in the facial image; then, based on the target region of interest in the plurality of facial images, the heart rate data of the user is determined. Compared with the related non-contact heart rate monitoring technology, which determines the region of interest based on preset rules, the present application determines the target region of interest in the facial image through a preset target prompt word and a pre-trained multimodal language model, and does not need to manually adjust parameters according to the use environment. The region of interest in the facial image can be dynamically evaluated, and the adaptability to the application scenario is stronger, so that a more accurate target region of interest can be obtained, and then more accurate heart rate data can be obtained, which solves the problem of poor accuracy of heart rate monitoring in the related non-contact heart rate monitoring technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of a non-contact user heart rate monitoring method provided in an embodiment of the present application; Figure 2 A flowchart of another non-contact user heart rate monitoring method provided in an embodiment of the present application; Figure 3 is a structural schematic diagram of a non-contact user heart rate monitoring device provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0017] The following, in conjunction with the accompanying drawings, describes in detail the contactless user heart rate monitoring method, device, electronic device and storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0018] Figure 1 A non-contact user heart rate monitoring method provided by an embodiment of the present invention is shown. The method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: S102: Acquire multiple facial images of the user.

[0019] In practical applications, the multiple facial images obtained may be continuous frames extracted from a facial video in chronological order, or may be a sequence of static images continuously shot at set time intervals, and there is no specific limitation on this.

[0020] S104: Determine a target region of interest in the plurality of face images based on a preset target prompt word and a pre-trained multimodal language model.

[0021] S106: Determine the user's heart rate data based on the target region of interest in the multiple face images.

[0022] The target prompt word is used to enable the pre-trained multimodal language model to screen the area in the face image to output the area of ​​interest in the face image.

[0023] In practical applications, the multimodal language model usually obtained has already been trained, and the obtained multimodal language model can be directly used as the above-mentioned pre-trained multimodal language model. Of course, it can also be trained again based on natural language knowledge related to rPPG to obtain the above-mentioned pre-trained multimodal language model.

[0024] Specifically, the pre-trained multimodal language model can be a multimodal large language model (Large Language Models, LLMs). For example, the natural language processing model GPT-4V based on deep learning. It can process multiple data types such as text and images at the same time. It has the ability to understand images and filter information, so that it can accurately identify and process complex visual factors in facial images. The pre-trained multimodal language model can learn the characteristics of different individuals and better handle individual differences caused by factors such as age and gender, so as to adapt to complex application environments.

[0025] The non-contact user heart rate monitoring method provided by the embodiment of the present invention first obtains multiple face images of the user; then determines the target region of interest in the multiple face images based on a preset target prompt word and a pre-trained multimodal language model; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the face image to output the region of interest in the face image; then, based on the target region of interest in the multiple face images, the user's heart rate data is determined. Compared with the related non-contact heart rate monitoring technology, which determines the region of interest based on preset rules, the present application determines the target region of interest in the face image through preset target prompt words and a pre-trained multimodal language model, and does not need to manually adjust parameters according to the use environment. It can dynamically evaluate the region of interest in the face image, and has stronger adaptability to application scenarios, so that a more accurate target region of interest can be obtained, and then more accurate heart rate data can be obtained, which solves the problem of poor accuracy of heart rate monitoring in the related non-contact heart rate monitoring technology.

[0026] In one implementation, the target prompt word includes: a first prompt word, and / or a second prompt word; wherein the first prompt word is related to the lighting condition of the area in the face image; and the second prompt word is related to the occlusion condition of the area in the face image.

[0027] Specifically, the first prompt word may include first information and first indication information related to the lighting condition of the area in the face image; the second prompt word may include second information and second indication information related to the occlusion condition of the area in the face image; the first indication information is used to instruct the pre-trained multimodal language model to convert the first information into third information corresponding to rPPG; the second indication information is used to instruct the pre-trained multimodal language model to convert the second information into fourth information corresponding to rPPG.

[0028] Considering that the lighting conditions in the actual application environment may be complex and changeable, such as the alternation of natural light during the day and night, the diverse artificial lighting, and the changes in the positional relationship between the light source and the objects in the environment, complex shadow patterns will be formed in the face area. In addition, if the light intensity is too strong, the video frame will be overexposed, and if it is too weak, the total amount of light penetrating the skin will be reduced, resulting in a lower intensity of the extracted physiological signal sequence. Targetedly, the first prompt word is preset so that the pre-trained multimodal language model can analyze the lighting conditions of the area in the face image, automatically filter out the overexposed or insufficiently lit areas, and select areas with more stable lighting, so as to achieve dynamic evaluation of the area of ​​interest, and thus obtain more accurate and reliable heart rate data.

[0029] It also takes into account the occlusion caused by hair and beards in different degrees in the actual application environment, as well as the occlusion caused by head movement. Targetedly, the second prompt word is preset to enable the pre-trained multimodal language model to analyze the occlusion of the area in the face image, automatically identify and select the optimal area, and realize dynamic evaluation of the area of ​​interest, thereby obtaining more accurate heart rate data. Compared with the relevant technology, since existing studies have shown that the forehead area can provide the most stable and effective signal extraction in the entire face area, the forehead area is taken as the area of ​​interest. Since the forehead area is affected by the occlusion of the hair, the quality of the physiological signal sequence is reduced, and even the physiological signal sequence is lost, which ultimately causes a decrease in the accuracy of heart rate monitoring. This embodiment takes into account that the area in the face image is easily affected by occlusion, can effectively deal with the occlusion problem, and improves the accuracy and reliability of the obtained physiological signal sequence.

[0030] Optionally, if the target prompt word includes the first prompt word, the area in the face image with the best lighting condition (considering the rPPG technical requirements) is screened out as the area of ​​interest, that is, the target area of ​​interest; if the target prompt word includes the first prompt word, the area in the face image with the best occlusion condition (considering the rPPG technical requirements) is screened out as the area of ​​interest, that is, the target area of ​​interest; if the target prompt word includes the first prompt word and the second prompt word, the area in the face image with the best occlusion condition (considering the rPPG technical requirements) and the best lighting condition (considering the rPPG technical requirements) is screened out as the target area of ​​interest.

[0031] In addition, before the above S104, different multimodal large models (determine a pre-trained multimodal language model from a preset plurality of pre-trained multimodal language models) can be selected to analyze the facial image according to the complexity of the usage scenario and the computing power of the hardware device.

[0032] Exemplarily, the first prompt word may be “Analyze this face image and consider the rPPG technical requirements. Identify areas with moderate brightness and few shadows. The ideal region of interest should have uniform lighting and avoid overly bright or dark parts”, prompting the pre-trained multimodal language model to consider brightness and shadow distribution, and filter out areas with higher light intensity as regions of interest. The second prompt word may be “Analyze this face image and consider the rPPG technical requirements. Identify areas with rich blood flow such as the forehead and cheeks. Avoid facial areas obscured by hair and beard”, which may enable the pre-trained multimodal language model to filter out areas not obscured by hair and beard as regions of interest. It should be noted that the limitations of the first prompt word and the second prompt word are only examples, and no specific limitations are imposed on the first prompt word and the second prompt word.

[0033] In one implementation, if the target prompt word includes the first prompt word and the second prompt word; based on the preset target prompt word and the pre-trained multimodal language model, the target region of interest in the multiple face images is determined (i.e., S104), and the following steps A1 to A3 may be specifically performed: Step A1: input a preset first prompt word and a face image into a pre-trained multimodal language model to obtain a first region of interest in the face image output by the pre-trained multimodal language model.

[0034] Step A2: input a preset second prompt word and a face image into a pre-trained multimodal language model, and obtain a second region of interest in the face image output by the pre-trained multimodal language model.

[0035] Step A3: determining an overlapping area between the first region of interest and the second region of interest in the plurality of face images, and taking the overlapping area as a target region of interest in the face image.

[0036] For example, the target prompt word can be T={t 1 ,t 2 ,…t n}, t i represents the i-th word or character, n is the length of the text; the first prompt word can be T 1 and the second prompt word can be T 2 The target prompt word realizes the injection of rPPG domain knowledge as the control instruction of the multimodal large model. 1 And the face image I is input into the multimodal large model, that is, the input of the multimodal large model can be expressed as: X input ={T 1 , I}. The pre-trained multimodal language model is used for input X inputProcessing is performed to output the first region of interest. Specifically, the bounding box corresponding to the first region of interest is [a1, b1, a2, b2], where (a1, b1) and (a2, b2) represent the coordinates of the upper left corner and the lower right corner of the bounding box, respectively, and the corresponding mask is recorded as Mask1. Any pixel point (a, b) contained in Mask1 satisfies the conditions: a1≤a≤a2 and b1≤b≤b2. The second prompt word T2 and the face image I are input into the multimodal large model, and the second region of interest is output. Specifically, the bounding box corresponding to the second region of interest is [c1, d1, c2, d2], where (c1, d1) and (c2, d2) represent the coordinates of the upper left corner and the lower right corner of the bounding box, respectively, and the corresponding mask is recorded as Mask2. Any pixel point (c, d) contained in Mask2 satisfies the conditions: c1≤c≤c2 and d1≤d≤d2. Mask1∩Mask2, that is, determine the overlapping area Mask3, and the overlapping area Mask3 is the target region of interest. It should be noted that the above definition of the bounding box and the mask is only an example and is not a specific limitation.

[0037] In this embodiment, through the preset first prompt word related to the lighting condition of the area in the face image and the preset second prompt word related to the occlusion condition of the area in the face image, the pre-trained multimodal language model can intelligently evaluate the lighting condition and occlusion condition of the face image to ensure the selection of the optimal skin area (target area of ​​interest), thereby improving the accuracy of heart rate monitoring.

[0038] In one implementation, obtaining multiple facial images of the user (ie, S102) may be specifically performed as follows: Steps B1 to B2: Step B1, obtaining video data of the user collected by the camera module.

[0039] Optionally, the camera module may be a separate camera or a camera module embedded in the electronic device.

[0040] Step B2, performing face detection on video frames in the video data using a pre-trained face detection model, determining face regions in the video frames of the video data, and obtaining multiple face images of the user.

[0041] Optionally, different training data may be selected to train the face detection model according to actual conditions, and the face detection model may be simplified and the processing speed may be optimized according to the complexity of the usage scenario.

[0042] Specifically, the face detection model can capture the image of the face area in the video frame, thereby obtaining multiple face images of the user. The face detection model can be a multi-task cascaded convolutional neural network (MTCNN), a face detection framework RetinaFace, etc.

[0043] In one implementation, based on the target region of interest in the multiple face images, determining the user's heart rate data (ie, step S106) can be specifically performed as follows: step C1 to step C2: Step C1, determining a physiological signal sequence of a user based on a target region of interest in a plurality of face images.

[0044] Specifically, the physiological signal sequence may be determined based on the pixel value of the target region of interest of each face image in the plurality of face images.

[0045] Specifically, the average value of the pixel values ​​of the target region of interest in each face image can be calculated, and the average value can be used as the physiological information. Then, the physiological information corresponding to multiple face images can be connected in time sequence to obtain the original physiological signal sequence.

[0046] Continuing with the above example, we can calculate the target regions of interest in multiple face images frame by frame to obtain a set of target regions of interest arranged in time, Mask. c,i =(Mask c,1 , …, Mask c,T ), where C includes three color channels, RGB, and T is the number of frames of the face image. Then, calculate the Mask c,1 The average value of all pixel values ​​in r c,1 , ..., calculate Mask c,T The average value of all pixel values ​​in r c,T , get the original physiological signal sequence r c,i =(r c,1 ,…,r c,T ), where i is any value of the number of frames of the face image, the range of i is [1, T], C∈{R, G, B}, that is, the original physiological signal sequence r c,i Including the time series corresponding to the R color channel, the time series corresponding to the G color channel and the time series corresponding to the B color channel.

[0047] Afterwards, the above original physiological signal sequence is processed to obtain a physiological signal sequence. Optionally, based on the original physiological signal sequence, according to the actual situation, any RGB time series fusion method can be selected to determine the physiological signal sequence. For example: when the head movement is small and the lighting conditions are relatively stable, the green channel in the original physiological signal sequence, that is, the time series corresponding to the R color channel can be used as the physiological signal sequence; if the head movement is large, the different color channels of the original physiological signal sequence can be weighted and superimposed according to the preset rules, for example: the original physiological signal sequence r c,i Including the time series corresponding to the three color channels of R, G and B. First, these channels are projected in the following way: the first signal is generated by using the difference between the time series corresponding to the G color channel and the time series corresponding to the B color channel (the projection vector is [0,1,-1]); secondly, the time series corresponding to the R color channel, the time series corresponding to the G color channel and the time series corresponding to the B color channel are linearly combined according to the formula -2×R+G+B (the projection vector is [-2,1,1]) to generate the second signal. Then, the ratio of the standard deviation of the first signal to the second signal is calculated (as the adjustment parameter α) to dynamically adjust the weight between the two signals, perform linear weighted fusion, and finally generate the target physiological signal sequence. The adjustment parameter is used to balance the amplitude differences of these signals, thereby improving the accuracy of the extracted physiological signal sequence.

[0048] Step C2, determining the user's heart rate data based on the user's physiological signal sequence.

[0049] In one implementation, based on the physiological signal sequence of the user, determining the user's heart rate data (ie, step C2) may be specifically performed as follows: step c1 to step c2: Step c1, performing bandpass filtering on the physiological signal sequence to obtain a heartbeat signal sequence of a preset frequency.

[0050] Specifically, a Butterworth filter may be used to set the cutoff frequency range within the normal human heartbeat range, that is, 0.75-2.5 Hz.

[0051] Step c2, performing Fourier transform processing on the heartbeat signal sequence of the preset frequency to obtain the user's heart rate data.

[0052] Specifically, after Fourier transform processing, the frequency value corresponding to the peak value of the frequency domain amplitude is selected as the heart rate, thereby obtaining the user's heart rate data.

[0053] Figure 2 FIG. 1 is a flow chart of another non-contact user heart rate monitoring method provided in an embodiment of the present application. Figure 2 As shown, the method includes: Step 202: Acquire video data of the user collected by the camera module.

[0054] Step 204 , performing face detection on the video frames in the video data using a pre-trained face detection model, determining the face regions in the video frames of the video data, and obtaining multiple face images of the user.

[0055] Step 206: Input the preset first prompt word and the face image into the pre-trained multimodal language model to obtain a first region of interest in the face image output by the pre-trained multimodal language model.

[0056] Among them, the target prompt word includes: a first prompt word, and / or a second prompt word; the target prompt word is used to enable the pre-trained multimodal language model to screen the area in the face image to output the area of ​​interest in the face image; the first prompt word includes first information and first indication information related to the lighting condition of the area in the face image; the second prompt word includes second information and second indication information related to the occlusion condition of the area in the face image; the first indication information is used to instruct the pre-trained multimodal language model to convert the first information into the third information corresponding to the rPPG; the second indication information is used to instruct the pre-trained multimodal language model to convert the second information into the fourth information corresponding to the rPPG.

[0057] Step 208: input the preset second prompt word and the face image into the pre-trained multimodal language model to obtain a second region of interest in the face image output by the pre-trained multimodal language model.

[0058] Step 210: determine the overlapping area between the first region of interest and the second region of interest in the plurality of face images, and use the overlapping area as the target region of interest in the face image.

[0059] Step 212: determining a physiological signal sequence based on the pixel values ​​of the target region of interest of each face image in the plurality of face images.

[0060] Step 214: bandpass filter the physiological signal sequence to obtain a heartbeat signal sequence of a preset frequency.

[0061] Step 216: Perform Fourier transform processing on the heartbeat signal sequence of the preset frequency to obtain the user's heart rate data.

[0062] The specific process of the above steps 202 to 216 has been described in detail in the above embodiment and will not be repeated here.

[0063] In this embodiment, multiple facial images of the user are first obtained; then, based on a preset target prompt word and a pre-trained multimodal language model, a target region of interest in the multiple facial images is determined; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the facial image to output the region of interest in the facial image; then, based on the target region of interest in the multiple facial images, the user's heart rate data is determined. Compared with the related non-contact heart rate monitoring technology, which determines the region of interest based on preset rules, the present application determines the target region of interest in the facial image through preset target prompt words and a pre-trained multimodal language model, and does not need to manually adjust parameters according to the use environment. The region of interest in the facial image can be dynamically evaluated, and the adaptability to the application scenario is stronger, so that a more accurate target region of interest can be obtained, and then more accurate heart rate data can be obtained, which solves the problem of poor accuracy of heart rate monitoring in the related non-contact heart rate monitoring technology.

[0064] Corresponding to the non-contact user heart rate monitoring method provided in the above embodiment, based on the same technical concept, the embodiment of the present invention also provides a non-contact user heart rate monitoring device, Figure 3 is a schematic diagram of the structure of a non-contact user heart rate monitoring device according to an embodiment of the present invention, wherein the non-contact user heart rate monitoring device is used to perform Figure 1 to Figure 2 A non-contact user heart rate monitoring method described, such as Figure 3 As shown, the non-contact user heart rate monitoring device includes: an acquisition module 310 , a first determination module 320 and a second determination module 330 .

[0065] An acquisition module 310 is used to acquire multiple facial images of a user; A first determination module 320 is used to determine a target region of interest in a plurality of face images based on a preset target prompt word and a pre-trained multimodal language model; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the face image to output the region of interest in the face image; The second determination module 330 is used to determine the user's heart rate data based on the target region of interest in the multiple face images.

[0066] In one implementation, the target prompt word includes: a first prompt word, and / or a second prompt word; wherein the first prompt word is related to the lighting condition of the area in the face image; and the second prompt word is related to the occlusion condition of the area in the face image.

[0067] In one implementation, the first determining module 320 is specifically configured to: Inputting a preset first prompt word and a face image into a pre-trained multimodal language model, and obtaining a first region of interest in the face image output by the pre-trained multimodal language model; Inputting a preset second prompt word and a face image into a pre-trained multimodal language model to obtain a second region of interest in the face image output by the pre-trained multimodal language model; An overlapping area between a first region of interest and a second region of interest in a plurality of face images is determined, and the overlapping area is used as a target region of interest in the face image.

[0068] In one implementation, the acquisition module 310 is specifically configured to: Acquire video data of the user collected by the camera module; The pre-trained face detection model is used to perform face detection on the video frames in the video data, and the face regions in the video frames of the video data are determined to obtain multiple face images of the user.

[0069] In one implementation, the second determining module 330 includes: A first determining unit, configured to determine a physiological signal sequence of the user based on the target region of interest in the plurality of face images; The second determining unit is used to determine the heart rate data of the user based on the physiological signal sequence of the user.

[0070] In one implementation, the second determining unit is specifically configured to: Performing bandpass filtering on the physiological signal sequence to obtain a heartbeat signal sequence of a preset frequency; The heartbeat signal sequence of the preset frequency is processed by Fourier transform to obtain the user's heart rate data.

[0071] In this embodiment, multiple facial images of the user are first obtained; then, based on a preset target prompt word and a pre-trained multimodal language model, a target region of interest in the multiple facial images is determined; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the facial image to output the region of interest in the facial image; then, based on the target region of interest in the multiple facial images, the user's heart rate data is determined. Compared with the related non-contact heart rate monitoring technology, which determines the region of interest based on preset rules, the present application determines the target region of interest in the facial image through preset target prompt words and a pre-trained multimodal language model, and does not need to manually adjust parameters according to the use environment. The region of interest in the facial image can be dynamically evaluated, and the adaptability to the application scenario is stronger, so that a more accurate target region of interest can be obtained, and then more accurate heart rate data can be obtained, which solves the problem of poor accuracy of heart rate monitoring in the related non-contact heart rate monitoring technology.

[0072] Those skilled in the art should understand that the above-mentioned non-contact user heart rate monitoring device can be used to implement the non-contact user heart rate monitoring method mentioned above, and the detailed description thereof should be similar to the description of the method part mentioned above, and will not be repeated here to avoid repetition.

[0073] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to perform the above-mentioned non-contact user heart rate monitoring method. Figure 4 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call a computer program stored in the memory 430 and executable on the processor 410 to perform the following steps: Get multiple facial images of the user; Based on a preset target cue word and a pre-trained multimodal language model, a target region of interest in a plurality of face images is determined; wherein the target cue word is used to enable the pre-trained multimodal language model to screen the region in the face image to output the region of interest in the face image; Based on target regions of interest in multiple face images, the user's heart rate data is determined.

[0074] In this embodiment, multiple facial images of the user are first obtained; then, based on a preset target prompt word and a pre-trained multimodal language model, a target region of interest in the multiple facial images is determined; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the facial image to output the region of interest in the facial image; then, based on the target region of interest in the multiple facial images, the user's heart rate data is determined. Compared with the related non-contact heart rate monitoring technology, which determines the region of interest based on preset rules, the present application determines the target region of interest in the facial image through preset target prompt words and a pre-trained multimodal language model, and does not need to manually adjust parameters according to the use environment. The region of interest in the facial image can be dynamically evaluated, and the adaptability to the application scenario is stronger, so that a more accurate target region of interest can be obtained, and then more accurate heart rate data can be obtained, which solves the problem of poor accuracy of heart rate monitoring in the related non-contact heart rate monitoring technology.

[0075] The specific execution steps can refer to the various steps of the above-mentioned non-contact user heart rate monitoring method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0076] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals or other devices except terminals.

[0077] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0078] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0079] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0080] An embodiment of the present application also provides a storage medium, on which computer executable instructions are stored. When the computer executable instructions are executed by a processor, the various processes of the above-mentioned non-contact user heart rate monitoring method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0081] The processor is the processor in the electronic device described in the above embodiment. The storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0082] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned non-contact user heart rate monitoring method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0083] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0084] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the various processes of the above-mentioned non-contact user heart rate monitoring method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0085] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include multitasking and parallel processing according to the functions involved, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0087] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A non-contact user heart rate monitoring method, characterized in that: The method comprises: Get multiple facial images of the user; Based on a preset target prompt word and a pre-trained multimodal language model, determining a target region of interest in the plurality of face images; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the face image to output the region of interest in the face image; Based on the target region of interest in the plurality of face images, the heart rate data of the user is determined.

2. The method according to claim 1, characterized in that: The target prompt word includes: a first prompt word, and / or a second prompt word; wherein, the first prompt word is related to the lighting condition of the area in the face image; and the second prompt word is related to the occlusion condition of the area in the face image.

3. The method according to claim 2, characterized in that The step of determining the target region of interest in the plurality of face images based on the preset target prompt words and the pre-trained multimodal language model includes: Inputting the preset first prompt word and the face image into the pre-trained multimodal language model, and obtaining a first region of interest in the face image output by the pre-trained multimodal language model; Inputting the preset second prompt word and the face image into the pre-trained multimodal language model, and obtaining a second region of interest in the face image output by the pre-trained multimodal language model; An overlapping area between the first region of interest and the second region of interest in a plurality of the face images is determined, and the overlapping area is used as the target region of interest in the face image.

4. The method according to claim 1, characterized in that: The step of obtaining multiple facial images of a user includes: Acquire video data of the user collected by a camera module; Face detection is performed on the video frames in the video data using a pre-trained face detection model to determine the face area in the video frames of the video data, so as to obtain a plurality of face images of the user.

5. The method according to claim 1, characterized in that The step of determining the heart rate data of the user based on the target region of interest in the plurality of face images includes: Determining a physiological signal sequence of the user based on the target region of interest in the plurality of face images; Based on the physiological signal sequence of the user, heart rate data of the user is determined.

6. The method according to claim 5, characterized in that The step of determining the heart rate data of the user based on the physiological signal sequence of the user comprises: Performing bandpass filtering on the physiological signal sequence to obtain a heartbeat signal sequence of a preset frequency; Perform Fourier transform processing on the heartbeat signal sequence of the preset frequency to obtain the heart rate data of the user.

7. A non-contact user heart rate monitoring device, characterized in that: The device comprises: An acquisition module, used to acquire multiple face images of the user; A first determination module is used to determine a target region of interest in the plurality of face images based on a preset target prompt word and a pre-trained multimodal language model; wherein the target prompt word is used to enable the pre-trained multimodal language model to screen the region in the face image to output the region of interest in the face image; The second determination module is used to determine the heart rate data of the user based on the target area of ​​interest in the multiple face images.

8. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the non-contact user heart rate monitoring method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the computer-executable instructions enable a computer to execute the non-contact user heart rate monitoring method as described in any one of claims 1-6.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the non-contact user heart rate monitoring method as described in any one of claims 1 to 6.

Images