Vital sign data detection method and apparatus, electronic device and storage medium

Abstract

The present application provides a vital sign data detection method and apparatus, an electronic device and a storage medium. The method comprises: acquiring multiple continuous frames of facial images of a user; on the basis of the multiple frames of facial images, obtaining multiple sets of feature data of a preset area of interest of the face of the user; on the basis of the multiple sets of feature data, obtaining remote photoplethysmographic waves corresponding to the feature data; inputting the remote photoplethysmographic waves into a pre-trained calibration model, to obtain calibrated remote photoplethysmographic waves output by the calibration model; inputting the calibrated remote photoplethysmographic waves into a pre-trained bidirectional long- and short-term memory network model, to obtain vital sign data output by the bidirectional long- and short-term memory network model. This makes it possible to detect vital sign data by non-contact, low-cost means, while ensuring that accurate vital sign data is obtained.

Description

Vital sign data detection method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310950855X, filed on July 31, 2023, entitled “Method, device, electronic device and storage medium for detecting vital signs data,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the technical field of vital sign data detection, and in particular to a vital sign data detection method, device, electronic device and storage medium. Background Art

[0004] As society progresses, users are paying more and more attention to their own health. During the application process, users' vital signs data can be used to monitor their own health.

[0005] In related technologies, users can use wearable electronic devices or medical testing equipment to monitor their vital signs. However, using wearable electronic devices to monitor vital signs often increases user costs. Using medical testing equipment to monitor vital signs requires contact or invasive testing, placing a psychological burden on users.

[0006] Therefore, finding a non-contact and cost-effective method for detecting vital signs data has become a current research hotspot.

[0007] Summary of the Invention

[0008] The present application provides a method, device, electronic device and storage medium for detecting vital signs data, which realize the detection of vital signs data in a non-contact and low-cost manner.

[0009] The present application provides a method for detecting vital sign data, the method comprising: acquiring multiple consecutive frames of facial images of a user; obtaining multiple sets of feature data of a preset region of interest of the user's face based on the multiple frames of facial images; obtaining remote photoplethysmography corresponding to the feature data based on the multiple sets of feature data; inputting the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmography; and inputting the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0010] According to a vital sign data detection method provided by the present application, the calibration model is trained in the following manner: synchronously obtaining multiple frames of continuous facial image samples of a user and multiple groups of contact photoplethysmography samples of the user, wherein the facial image samples and the contact photoplethysmography samples have a corresponding relationship in the time dimension; based on the multiple frames of the facial image samples, multiple groups of feature data samples of a preset region of interest of the user's face are obtained; based on the multiple groups of the feature data samples, remote photoplethysmography samples corresponding to the feature data samples are obtained; based on the remote photoplethysmography samples and the contact photoplethysmography samples that have a corresponding relationship with the remote photoplethysmography samples in the time dimension, the calibration model is trained to obtain a trained calibration model.

[0011] According to a vital sign data detection method provided by the present application, the feature data includes the channel value of any one of the RGB channels of the facial image; the remote photoplethysmography corresponding to the feature data is obtained based on multiple sets of the feature data, specifically including: constructing a channel value curve formed by the channel values ​​and the time information based on the multiple sets of channel values ​​and the time information corresponding to each of the channel values; and obtaining the remote photoplethysmography corresponding to the feature data based on the channel value curve.

[0012] According to a vital sign data detection method provided by the present application, the characteristic data samples include channel sample values ​​of any one channel in the RGB channel samples of the facial image sample; the remote photoplethysmography samples corresponding to the characteristic data samples are obtained based on multiple groups of the characteristic data samples, specifically including: based on the multiple groups of the channel sample values ​​and the time information corresponding to each of the channel sample values, constructing a channel sample value curve formed by the channel sample values ​​and the time information; and obtaining the remote photoplethysmography samples corresponding to the characteristic data samples based on the channel sample value curve.

[0013] According to a vital sign data detection method provided by the present application, the vital sign data includes multiple vital sign data; after obtaining the vital sign data output by the bidirectional long short-term memory network model, the method further includes: obtaining a comprehensive score of the vital sign data of the user's vital sign data based on the multiple vital sign data; determining a vital sign level corresponding to the comprehensive score of the vital sign data based on the comprehensive score of the vital sign data and a pre-constructed first mapping table, wherein the first mapping table includes a correspondence between different comprehensive scores of vital sign data and different vital sign levels; and providing the user with a dietary recipe corresponding to the vital sign level based on the vital sign level.

[0014] According to a vital signs data detection method provided by the present application, the method of obtaining a comprehensive score of the vital signs data of the user's vital signs data based on multiple vital signs data specifically includes: obtaining a vital signs data score for each vital signs data based on each vital signs data and a standard value of the vital signs data corresponding to each vital signs data; separately determining a weight value for each vital signs data; and obtaining a comprehensive score of the vital signs data of the user's vital signs data based on a weighted summation of the vital signs data score for each vital signs data and the weight value of each vital signs data.

[0015] According to a vital sign data detection method provided in the present application, the method provides the user with a dietary recipe corresponding to the vital sign level based on the vital sign level, specifically including: pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different vital sign levels and different dietary recipe sets, wherein the dietary recipe set includes multiple dietary recipes; based on the vital sign level, determining a dietary recipe set corresponding to the vital sign level; obtaining the user's taste preference information; and based on the taste preference information, determining the dietary recipe in the dietary recipe set that matches the taste preference information.

[0016] According to a vital sign data detection method provided by the present application, after providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level, the method also includes: determining the recipe calorie value corresponding to the dietary recipe based on the dietary recipe; determining the standard reserved calorie value corresponding to the physical sign level based on the physical sign level; determining the calorie value that the user should consume based on the standard reserved calorie value and the recipe calorie value; and determining an exercise plan that matches the physical sign level and the dietary recipe of the user based on the calorie value that should be consumed.

[0017] The present application also provides a vital sign data detection device, which includes: an acquisition module for acquiring multiple consecutive frames of facial images of a user; a feature data determination module for obtaining multiple sets of feature data of a preset region of interest of the user's face based on multiple frames of the facial images; a remote photoplethysmography module for determining a remote photoplethysmography corresponding to the feature data based on multiple sets of the feature data; a processing module for inputting the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmography; and a generation module for inputting the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0018] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the vital sign data detection method as described above is implemented.

[0019] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting vital sign data as described above is implemented.

[0020] The present application also provides a computer program product, comprising a computer program, which implements any of the above-described vital sign data detection methods when executed by a processor.

[0021] The vital signs data detection method, device, electronic device and storage medium provided by the present application obtain multiple sets of feature data of a preset region of interest of the user's face based on a continuous multi-frame facial image of the user; obtain remote photoplethysmography corresponding to the feature data based on the multiple sets of feature data; then input the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, and input the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital signs data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmography is obtained through calibration, it can be ensured that the calibrated remote photoplethysmography is closer to the real remote photoplethysmography, thereby ensuring that the ultimately obtained vital signs data is more accurate. This ensures that, under the premise of ensuring that the vital signs data is accurately obtained, the detection of vital signs data can be achieved in a non-contact, low-cost manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] FIG1 is a flow chart of a method for detecting vital signs data provided by the present application;

[0024] FIG2 is a flow chart of the training and calibration model provided in this application;

[0025] FIG3 is a second flow chart of the vital sign data detection method provided by the present application;

[0026] FIG4 is a third flow chart of the vital sign data detection method provided by the present application;

[0027] FIG5 is a schematic structural diagram of a vital sign data detection device provided by the present application;

[0028] FIG6 is a schematic structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0030] In one embodiment, with the popularization of smart mobile terminal devices, such as mobile phones, tablets, smart home central control screens, laptops, smart magic mirrors, smart refrigerators, etc., and these smart mobile terminal devices are equipped with image acquisition devices, the vital signs data detection solution based on multi-frame images can be realized without increasing any hardware costs. Through the image acquisition device carried by the smart mobile terminal device itself, such as a camera, remote photoelectric volume pulse wave technology is used to achieve non-contact vital signs data including heart rate, blood pressure, blood oxygen, blood sugar and other indicators in real time and continuous detection.

[0031] The executor of the vital signs data detection method provided in this application can be a smart mobile terminal device. It has no special requirements for the hardware and system of the smart mobile terminal device, no strict requirements for the camera hardware configuration, and can be upward compatible. The method of obtaining the user's vital signs indicators can be an edge acquisition method or a cloud acquisition method. In another example, the vital signs data can be associated with the user's dietary health, exercise management, body management and other scenarios in real time, thereby providing users with active, non-sensitive, customized and personalized solutions.

[0032] It should be noted that this solution has the advantages of low detection hardware cost, flexible, real-time and continuous detection, rich detection indicators, and is associated with active, unconscious and personalized service scenarios such as user healthy diet.

[0033] FIG1 is a flow chart of a method for detecting vital signs data provided in the present application.

[0034] In order to further introduce the vital sign data detection method provided by this application, it will be explained below with reference to FIG1 .

[0035] In an exemplary embodiment of the present application, as can be seen from FIG. 1 , the vital sign data detection method may include steps 110 to 150 , and each step will be described below.

[0036] In step 110, multiple frames of continuous facial images of the user are obtained.

[0037] In step 120, multiple sets of feature data of the user's preset region of interest are obtained based on multiple frames of facial images.

[0038] In one embodiment, a plurality of consecutive facial images of a user can be captured using an image capture device inherent in a smart mobile terminal device, and based on the plurality of consecutive facial images, multiple sets of feature data for preset facial regions of interest of the user can be obtained. The preset facial regions of interest are preset areas within a facial image that can be adjusted based on actual conditions, such as the forehead, cheeks, or mouth. In this embodiment, no specific limitation is imposed on the preset facial regions of interest. In this embodiment, by capturing a plurality of consecutive facial images of a user using an image capture device inherent in a smart mobile terminal device, facial images can be acquired without any additional hardware costs, laying the foundation for obtaining vital sign data based on facial images without incurring any costs.

[0039] In another example, the feature data does not contain data information for identifying the user, so the feature data cannot be restored to the user's face image and video, thus preventing the user's privacy from being leaked.

[0040] In step 130 , based on the multiple sets of feature data, remote photoplethysmography corresponding to the feature data is obtained.

[0041] In another exemplary embodiment of the present application, the feature data includes a channel value of any one of the RGB channels of the facial image, such as an R channel value, a G channel value, or a B channel value. Based on multiple sets of feature data, obtaining a remote photoplethysmography corresponding to the feature data can be achieved in the following manner:

[0042] Based on multiple groups of channel values ​​and time information corresponding to each channel value, construct a channel value curve formed by the channel values ​​and the time information;

[0043] Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

[0044] In one embodiment, since the multiple sets of channel values ​​are channel values ​​under continuous frames, the time information corresponding to each channel value can be obtained. Furthermore, based on the multiple sets of channel values ​​and the time information corresponding to each channel value, a channel value curve formed by the channel value and the time information can be constructed. In other words, the channel value curve can be obtained with the channel value as the horizontal coordinate and the time information as the vertical coordinate. During the application process, the remote photoplethysmography corresponding to the feature data can be obtained based on the channel value curve. Among them, the aforementioned channel values ​​can be determined based on the average value of the channel values ​​of each pixel point in the preset area of ​​interest of the face.

[0045] In step 140 , the remote photoplethysmography is input into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model.

[0046] In step 150 , the calibrated remote photoplethysmography is input into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0047] In one embodiment, the obtained remote photoplethysmography can be input into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model. The remote photoplethysmography is a photoplethysmography obtained in a non-contact manner. The remote photoplethysmography may not be accurate enough. Therefore, in order to improve the accuracy of the remote photoplethysmography obtained in a non-contact manner, the remote photoplethysmography needs to be calibrated. The calibration model is a network model, for example, a U-NET network model. The calibration model is used to calibrate the remote photoplethysmography to obtain a calibrated remote photoplethysmography. The calibration model is pre-trained based on a sample set.

[0048] During the application process, the remote photoplethysmography can be calibrated based on a pre-trained calibration model to obtain a calibrated remote photoplethysmography. Furthermore, the calibrated remote photoplethysmography is input into a pre-trained bidirectional long short-term memory network model, thereby obtaining vital signs data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmography is obtained through calibration, it can be ensured that the calibrated remote photoplethysmography is closer to the real remote photoplethysmography, thereby ensuring that the vital signs data finally obtained is more accurate. This ensures that, under the premise of ensuring that the vital signs data are accurately obtained, the detection of vital signs data can be achieved in a non-contact, low-cost manner.

[0049] In another example, the bidirectional long short-term memory network model can be trained based on the calibrated remote photoplethysmography sample set to obtain a trained bidirectional long short-term memory network model. In other words, the bidirectional long short-term memory network model is obtained by pre-training with the sample training set.

[0050] The method for detecting vital signs data provided by the present application obtains multiple sets of feature data of a user's preset region of interest based on a continuous multi-frame facial image of the user; obtains remote photoplethysmography corresponding to the feature data based on the multiple sets of feature data; then inputs the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, and inputs the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital signs data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmography is obtained through calibration, it can be ensured that the calibrated remote photoplethysmography is closer to the real remote photoplethysmography, thereby ensuring that the ultimately obtained vital signs data is more accurate. This ensures that, under the premise of ensuring that the vital signs data is accurately obtained, the detection of vital signs data can be achieved in a non-contact, low-cost manner.

[0051] FIG2 is a flow chart of the training and calibration model provided in this application.

[0052] The process of training and calibrating the model will be described below with reference to FIG2 .

[0053] In an exemplary embodiment of the present application, as can be seen from FIG. 2 , training the calibration model may include steps 210 to 240 , and each step will be described below.

[0054] In step 210 , a plurality of consecutive frames of facial image samples of the user and a plurality of groups of contact photoplethysmography samples of the user are synchronously acquired.

[0055] In one embodiment, multiple consecutive frames of facial image samples of the user and multiple groups of contact photoplethysmography samples of the user can be obtained synchronously. In other words, there is a correspondence between the multiple consecutive frames of facial image samples and the contact photoplethysmography samples in the time dimension.

[0056] In yet another embodiment, the contact photoplethysmography sample may be obtained based on a contact test between the detection device and the user.

[0057] In step 220, based on the multiple frames of facial image samples, multiple groups of feature data samples of the preset regions of interest of the user's face are obtained.

[0058] In step 230 , based on the multiple sets of characteristic data samples, remote photoplethysmography samples corresponding to the characteristic data samples are obtained.

[0059] In step 240 , a calibration model is trained based on the remote photoplethysmography samples and the contact photoplethysmography samples that correspond to the remote photoplethysmography samples in the time dimension to obtain a trained calibration model.

[0060] In one embodiment, multiple sets of feature data samples for a preset region of interest on the user's face can be obtained based on multiple frames of facial image samples. Since there is a temporal correspondence between the multiple consecutive frames of facial image samples and the contact photoplethysmography samples, and the multiple sets of feature data samples are obtained based on the multiple consecutive frames of facial image samples, there is also a temporal correspondence between the multiple sets of feature data samples and the contact photoplethysmography samples.

[0061] Furthermore, the calibration model can be trained based on the remote photoplethysmography samples and the contact photoplethysmography samples that correspond to the remote photoplethysmography samples in the time dimension to obtain a trained calibration model. Since the contact photoplethysmography samples are obtained based on the contact test between the detection device and the user, the contact photoplethysmography samples are more accurate. In the present application, the remote photoplethysmography samples can be calibrated with the contact photoplethysmography samples as the true value. In the present application, the calibration model is trained based on the above ideas to ensure that the remote photoplethysmography input into the trained calibration model can obtain a more accurate calibrated remote photoplethysmography.

[0062] It should be noted that face image samples and face images have the same physical meaning; the difference is that face image samples are training samples, while face images are the face images to be processed. Based on the same principle, the relationship between contact photoplethysmography samples and contact photoplethysmography, feature data samples and feature data, and remote photoplethysmography samples and remote photoplethysmography is similar to the relationship between face image samples and face images.

[0063] In another exemplary embodiment of the present application, the feature data samples may include channel sample values ​​of any one of the RGB channel samples of the facial image sample. Furthermore, based on multiple sets of feature data samples, obtaining remote photoplethysmography samples corresponding to the feature data samples (corresponding to step 230) may be implemented in the following manner:

[0064] Based on multiple groups of channel sample values ​​and time information corresponding to each channel sample value, construct a channel sample value curve formed by the channel sample values ​​and the time information;

[0065] Based on the channel sample value curve, a remote photoplethysmography sample corresponding to the characteristic data sample is obtained.

[0066] The principle of obtaining the remote photoplethysmography sample corresponding to the characteristic data sample is the same as or similar to the principle of obtaining the remote photoplethysmography corresponding to the characteristic data, and will not be described in detail in this embodiment.

[0067] FIG3 is a second flow chart of the vital sign data detection method provided in this application.

[0068] In order to further introduce the vital sign data detection method provided by this application, it will be explained below with reference to FIG3 .

[0069] In an exemplary embodiment of the present application, the vital sign data may include multiple vital sign data, such as heart rate, blood pressure, blood oxygen (PO2), standard deviation of sinus beat-to-beat interval (SDNN), blood sugar, respiratory rate, pressure value, glycated hemoglobin (A1C), etc. In this embodiment, the vital sign data is not specifically limited.

[0070] 3 , after obtaining the vital sign data output by the bidirectional long short-term memory network model (corresponding to step 150 ), the vital sign data detection method may further include steps 310 to 330 , each of which will be described below.

[0071] In step 310 , a comprehensive score of the user's vital sign data is obtained based on the plurality of vital sign data.

[0072] In one embodiment, the vital signs data detection method can also be implemented based on an APP installed on a smart mobile terminal device. Furthermore, multiple vital signs data can be obtained, and a comprehensive score of the user's vital signs data can be obtained based on the vital signs data. It can be understood that the comprehensive score of vital signs data is a score obtained by comprehensively considering each vital sign, thereby ensuring that the obtained comprehensive score of vital signs data is more accurate and reasonably matches the user's vital signs, thereby laying the foundation for providing the user with a reasonable dietary recipe.

[0073] In step 320 , the physical sign level corresponding to the physical sign data comprehensive score is determined based on the physical sign data comprehensive score and a pre-constructed first mapping table, wherein the first mapping table includes a correspondence between different physical sign data comprehensive scores and different physical sign levels.

[0074] In step 330 , based on the physical sign level, a dietary recipe corresponding to the physical sign level is provided to the user.

[0075] In one embodiment, a first mapping table can be pre-constructed, wherein the first mapping table includes a correspondence between different comprehensive scores of vital sign data and different vital sign levels. In one example, the comprehensive score A of vital sign data corresponds to the vital sign level a; the comprehensive score B of vital sign data corresponds to the vital sign level b. During the application process, when it is determined that the comprehensive score of the user's vital sign data is the comprehensive score A of the vital sign data, the user's vital sign level can be determined to be the vital sign level a. Furthermore, based on the vital sign level, a dietary recipe corresponding to the vital sign level can be provided to the user, wherein the dietary recipe can include the ingredients to be consumed and the intake of the corresponding ingredients. Through this embodiment, reasonable dietary recipes can be proactively provided to the user based on the obtained vital sign data.

[0076] In another exemplary embodiment of the present application, obtaining a comprehensive score of a user's vital sign data based on multiple vital sign data may be achieved in the following manner:

[0077] Based on each vital sign data and the vital sign data standard value corresponding to each vital sign data, a vital sign data score of each vital sign data is obtained;

[0078] Determine the weight value of each vital sign data separately;

[0079] A weighted sum is performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive vital sign data score of the user's vital sign data.

[0080] In one embodiment, the standard value of the vital sign data corresponding to each vital sign data may be obtained respectively, and the vital sign data score of the vital sign data may be obtained based on the vital sign data and the standard value of the vital sign data corresponding to the vital sign data.

[0081] Let's take blood pressure as an example. For example, if a user's blood pressure is m and the standard blood pressure value is M, then the blood pressure data score can be determined based on the difference between the standard blood pressure value M and blood pressure m. It's understood that the difference between the standard blood pressure value M and blood pressure m is negatively correlated with the blood pressure data score.

[0082] Furthermore, weight values ​​for each vital sign data item can be obtained. The weight values ​​can be determined based on the importance of the vital sign data item. The more important the vital sign data item is, the greater the corresponding weight value. In one example, blood pressure and weight are used as vital sign data items. Because blood pressure is more important than weight, the weight value corresponding to blood pressure is greater than the weight value corresponding to weight.

[0083] In another embodiment, a weighted summation can be performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive vital sign data score for the user's vital sign data. Continuing to use blood pressure and weight as an example, the vital sign data scores and weight values ​​of blood pressure and weight can be weighted summed to obtain a comprehensive vital sign data score for the user's vital sign data. This ensures that the obtained comprehensive vital sign data score comprehensively considers each vital sign data and the importance of each vital sign data, laying the foundation for recommending reasonable dietary recipes based on the comprehensive vital sign data score.

[0084] In another exemplary embodiment of the present application, based on the physical sign level, providing the user with a meal recipe corresponding to the physical sign level can be achieved in the following manner:

[0085] Pre-constructing a second mapping table, wherein the second mapping table includes correspondences between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes;

[0086] Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level;

[0087] Obtain user taste preference information;

[0088] Based on the taste preference information, a meal recipe matching the taste preference information is determined in the meal recipe collection.

[0089] In one embodiment, a second mapping table may be pre-constructed. The second mapping table may include correspondences between different physical sign levels and different dietary recipe sets. In one example, physical sign level a may correspond to dietary recipe set a'; and physical sign level b may correspond to dietary recipe set b'.

[0090] Furthermore, when it is determined that the user's physical sign level is physical sign level a, the dietary recipe set corresponding to the user may be determined to be a dietary recipe set a', wherein the dietary recipe set a' includes a plurality of dietary recipes.

[0091] In another embodiment, the user's taste preference information may be obtained, and further, based on the user's taste preference information, a dietary recipe matching the taste preference information may be determined from the dietary recipe collection. During application, the determined dietary recipe matching the taste preference information may be used as a dietary recipe recommended to the user.

[0092] In another embodiment, a dietary recipe that matches the taste preference information can be used as a dietary recipe recommended to the user. Through this embodiment, it can be ensured that the dietary recipe recommended to the user not only meets the user's physical sign level but also meets the user's taste preferences, thereby improving the user's experience and satisfaction.

[0093] FIG4 is a third flow chart of the vital sign data detection method provided in this application.

[0094] In another exemplary embodiment of the present application, in combination with Figure 4, after providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level, the vital sign data detection method can also include steps 410 to 440, and each step will be introduced below.

[0095] In step 410 , based on the meal recipe, a recipe calorie value corresponding to the meal recipe is determined.

[0096] In step 420 , based on the physical sign level, a standard conservation calorie value corresponding to the physical sign level is determined.

[0097] In one embodiment, the calorie value of the corresponding recipe in the dietary recipe recommended for the user can be determined. Furthermore, based on the physical sign level, a standard reserved calorie value corresponding to the physical sign level is determined. It is understandable that different physical sign levels can correspond to different standard reserved calorie values. When the user's physical sign level is higher, the corresponding standard consumption calorie value can be higher. The standard reserved calorie value is the calorie value that the user maintains the reasonable physical sign needs.

[0098] In step 430 , the calorie value that the user should consume is determined based on the standard stored calorie value and the recipe calorie value.

[0099] In step 440 , an exercise program matching the user's physical sign level and dietary recipe is determined based on the calorie value to be consumed.

[0100] In one embodiment, the user's caloric intake can be determined based on the difference between the recipe's caloric value and the standard stored caloric value. Furthermore, based on the caloric intake, an exercise plan can be determined that matches the user's physical level and dietary recipe, and the exercise plan can be proactively provided to the user. This ensures that the user's body shape is maintained through an exercise plan that matches the user's physical level and dietary recipe.

[0101] According to the above description, the vital signs data detection method provided by the present application obtains multiple sets of feature data of the user's preset area of ​​interest based on the user's continuous multi-frame facial images; obtains remote photoplethysmography corresponding to the feature data based on the multiple sets of feature data; then inputs the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, and inputs the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital signs data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmography is obtained through calibration, it can be ensured that the calibrated remote photoplethysmography is closer to the real remote photoplethysmography, thereby ensuring that the final vital signs data is more accurate. This ensures that, under the premise of ensuring that the vital signs data is accurately obtained, the detection of vital signs data can be achieved in a non-contact, low-cost manner.

[0102] Based on the same concept, the present application also provides a vital sign data detection device.

[0103] The vital sign data detection device provided in the present application is described below. The vital sign data detection device described below and the vital sign data detection method described above can refer to each other.

[0104] FIG5 is a schematic structural diagram of the vital sign data detection device provided in this application.

[0105] In an exemplary embodiment of the present application, as shown in Figure 5, the vital sign data detection device may include an acquisition module 510, a feature data determination module 520, a remote photoplethysmography determination module 530, a processing module 540 and a generation module 550. Each module will be introduced below.

[0106] The acquisition module 510 may be configured to acquire multiple consecutive frames of facial images of the user;

[0107] The feature data determination module 520 may be configured to obtain multiple sets of feature data of a preset region of interest of the user's face based on multiple frames of facial images;

[0108] The remote photoplethysmography determination module 530 may be configured to obtain a remote photoplethysmography corresponding to the characteristic data based on the plurality of sets of characteristic data;

[0109] The processing module 540 may be configured to input the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmography;

[0110] The generation module 550 may be configured to input the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0111] In an exemplary embodiment of the present application, the processing module 540 may train and obtain the calibration model in the following manner:

[0112] synchronously acquiring multiple consecutive frames of facial image samples of the user and multiple sets of contact photoplethysmography samples of the user, wherein the facial image samples and the contact photoplethysmography samples have a corresponding relationship in a time dimension;

[0113] Based on multiple frames of facial image samples, multiple sets of feature data samples of the user's preset area of ​​interest are obtained;

[0114] Based on the multiple sets of characteristic data samples, remote photoplethysmography samples corresponding to the characteristic data samples are obtained;

[0115] The calibration model is trained based on the remote photoplethysmography samples and the contact photoplethysmography samples corresponding to the remote photoplethysmography samples in the time dimension to obtain a trained calibration model.

[0116] In an exemplary embodiment of the present application, the feature data may include a channel value of any one of the RGB channels of the face image;

[0117] The remote photoplethysmography determination module 530 may obtain the remote photoplethysmography corresponding to the characteristic data based on multiple sets of characteristic data in the following manner:

[0118] Based on multiple groups of channel values ​​and time information corresponding to each channel value, construct a channel value curve formed by the channel values ​​and the time information;

[0119] Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

[0120] In an exemplary embodiment of the present application, the feature data sample may include a channel sample value of any one of the RGB channel samples of the face image sample;

[0121] The processing module 540 may obtain remote photoplethysmography samples corresponding to the characteristic data samples based on multiple sets of characteristic data samples in the following manner:

[0122] Based on multiple groups of channel sample values ​​and time information corresponding to each channel sample value, construct a channel sample value curve formed by the channel sample values ​​and the time information;

[0123] Based on the channel sample value curve, a remote photoplethysmography sample corresponding to the characteristic data sample is obtained.

[0124] In an exemplary embodiment of the present application, the vital sign data may include a plurality of vital sign data;

[0125] The generation module 550 may also be configured to:

[0126] Based on multiple vital sign data, a comprehensive score of the user's vital sign data is obtained;

[0127] Determining the physical sign level corresponding to the physical sign data comprehensive score based on the physical sign data comprehensive score and a pre-constructed first mapping table, wherein the first mapping table includes a correspondence between different physical sign data comprehensive scores and different physical sign levels;

[0128] Based on the physical sign level, the user is provided with a meal recipe corresponding to the physical sign level.

[0129] In an exemplary embodiment of the present application, the generation module 550 may also adopt the following method to obtain the comprehensive score of the user's vital sign data based on multiple vital sign data:

[0130] Based on each vital sign data and the vital sign data standard value corresponding to each vital sign data, a vital sign data score of each vital sign data is obtained;

[0131] Determine the weight value of each vital sign data separately;

[0132] A weighted sum is performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive vital sign data score of the user's vital sign data.

[0133] In an exemplary embodiment of the present application, the generation module 550 may also provide the user with a dietary recipe corresponding to the physical sign level based on the physical sign level in the following manner:

[0134] Pre-constructing a second mapping table, wherein the second mapping table includes correspondences between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes;

[0135] Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level;

[0136] Obtain user taste preference information;

[0137] Based on the taste preference information, a meal recipe matching the taste preference information is determined in the meal recipe collection.

[0138] In an exemplary embodiment of the present application, the generating module 550 may also be configured to:

[0139] Based on the dietary recipe, determining a recipe calorie value corresponding to the dietary recipe;

[0140] Based on the physical sign level, determining the standard conservation calorie value corresponding to the physical sign level;

[0141] Determine the calorie value that the user should consume based on the standard saved calorie value and the recipe calorie value;

[0142] Based on the calorie value that should be consumed, an exercise plan that matches the user's physical level and dietary recipe is determined.

[0143] Figure 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 6, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the vital sign data detection method, which includes: obtaining multiple consecutive frames of facial images of the user; obtaining multiple sets of feature data of a preset area of ​​interest of the user's face based on the multiple frames of facial images; obtaining remote photoplethysmography corresponding to the feature data based on the multiple sets of feature data; inputting the remote photoplethysmography into a pre-trained calibration model to obtain a calibrated remote photoplethysmography output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmography; inputting the calibrated remote photoplethysmography into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0144] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0145] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vital sign data detection method provided by the above methods, the method including: obtaining multiple frames of continuous facial images of the user; based on the multiple frames of facial images, obtaining multiple sets of feature data of a preset area of ​​interest of the user's face; based on the multiple sets of feature data, obtaining a remote photoplethysmogram corresponding to the feature data; inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; inputting the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0146] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vital sign data detection method provided by the above-mentioned methods, the method comprising: obtaining a plurality of consecutive frames of facial images of a user; obtaining a plurality of sets of feature data of a preset region of interest of the user's face based on the plurality of frames of facial images; obtaining a remote photoplethysmogram corresponding to the feature data based on the plurality of sets of feature data; inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; inputting the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0149] It should be further understood that although operations are described in a particular order in the drawings in the embodiments of the present application, this should not be construed as requiring that these operations be performed in the particular order shown or in a serial order, or that all of the illustrated operations be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting vital signs data, comprising: Obtain multiple consecutive frames of facial images of the user; Based on the multiple frames of the facial images, obtaining multiple sets of feature data of a preset region of interest of the user's face; Based on the plurality of sets of characteristic data, obtaining remote photoplethysmograms corresponding to the characteristic data; Inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; The calibrated remote photoplethysmogram is input into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

2. The method for detecting vital signs data according to claim 1, wherein: The calibration model is trained in the following way: Synchronously acquiring multiple consecutive frames of face image samples of the user and multiple groups of contact photoplethysmogram samples of the user, wherein the face image samples and the contact photoplethysmogram samples have a corresponding relationship in the time dimension; Based on the multiple frames of face image samples, multiple groups of feature data samples of the preset regions of interest of the user's face are obtained; Based on the plurality of sets of characteristic data samples, obtaining remote photoplethysmogram samples corresponding to the characteristic data samples; Based on the remote photoplethysmography samples and the contact photoplethysmography samples corresponding to the remote photoplethysmography samples in the time dimension, the calibration model is trained to obtain a trained calibration model.

3. The method for detecting vital signs data according to claim 1, wherein: The feature data includes a channel value of any one of the RGB channels of the face image; The obtaining, based on the plurality of sets of characteristic data, a remote photoplethysmogram corresponding to the characteristic data, specifically comprises: Based on multiple groups of channel values ​​and time information corresponding to each channel value, construct a channel value curve formed by the channel value and the time information; Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

4. The method for detecting vital signs data according to claim 2, wherein: The feature data sample includes a channel sample value of any channel in the RGB channel samples of the face image sample; The obtaining, based on the plurality of groups of characteristic data samples, remote photoplethysmogram samples corresponding to the characteristic data samples specifically includes: Based on the multiple groups of channel sample values ​​and the time information corresponding to each of the channel sample values, construct a channel sample value curve formed by the channel sample values ​​and the time information; Based on the channel sample value curve, a remote photoplethysmogram sample corresponding to the characteristic data sample is obtained.

5. The method for detecting vital signs data according to claim 1, wherein: The vital sign data includes a plurality of vital sign data; After obtaining the vital sign data output by the bidirectional long short-term memory network model, the method further includes: Based on the plurality of vital sign data, obtaining a comprehensive score of the vital sign data of the user; Based on the comprehensive score of the physical sign data and a pre-constructed first mapping table, determining the physical sign level corresponding to the comprehensive score of the physical sign data, wherein the first mapping table includes a correspondence between different comprehensive scores of the physical sign data and different physical sign levels; Based on the physical sign level, a dietary recipe corresponding to the physical sign level is provided to the user.

6. The method for detecting vital signs data according to claim 5, wherein: The obtaining of a comprehensive score of the vital sign data of the user's vital sign data based on the plurality of vital sign data specifically includes: Based on each of the vital sign data and the vital sign data standard value corresponding to each of the vital sign data, obtaining a vital sign data score for each of the vital sign data; Determining the weight value of each of the vital sign data respectively; Based on the vital sign data score of each of the vital sign data and the weight value of each of the vital sign data, a weighted sum is performed to obtain the vital sign data of the user. According to the comprehensive score.

7. The method for detecting vital signs data according to claim 5, wherein: The providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level specifically includes: Pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes; Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level; Acquiring the user's taste preference information; Based on the taste preference information, the meal recipe matching the taste preference information is determined in the meal recipe set.

8. The method for detecting vital signs data according to claim 5 or 7, wherein: After providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level, the method further includes: Based on the dietary recipe, determining a recipe calorie value corresponding to the dietary recipe; Based on the physical sign level, determining a standard preservation calorie value corresponding to the physical sign level; Determining the calorie value that the user should consume based on the standard saved calorie value and the recipe calorie value; Based on the calorie value to be consumed, an exercise plan matching the physical sign level and the dietary recipe of the user is determined.

9. A vital sign data detection device, comprising: An acquisition module is used to acquire multiple consecutive frames of facial images of the user; A feature data determination module is used to obtain multiple sets of feature data of a preset region of interest of the user's face based on multiple frames of the face image; Determine a remote photoplethysmography module, for obtaining a remote photoplethysmography corresponding to the characteristic data based on a plurality of sets of the characteristic data; A processing module, used for inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used for calibrating the remote photoplethysmogram; A generating module is used to input the calibrated remote photoplethysmogram into a pre-trained A good bidirectional long short-term memory network model is obtained to obtain the vital sign data output by the bidirectional long short-term memory network model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the vital sign data detection method as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the vital sign data detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the vital sign data detection method according to any one of claims 1 to 8 is implemented.

Images