Vital sign data detection method and device, electronic equipment and storage medium

Abstract

The invention provides a vital sign data detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring continuous multi-frame face images of a user; based on the multiple frames of face images, obtaining multiple groups of feature data of a preset face region of interest of the user; based on the multiple sets of feature data, remote photoelectric volume pulse waves corresponding to the feature data are obtained; inputting the remote photoelectric volume pulse wave into a pre-trained calibration model to obtain a calibrated remote photoelectric volume pulse wave output by the calibration model; and inputting the calibrated remote photoelectric volume pulse wave into a pre-trained bidirectional long-short-term memory network model to obtain vital sign data output by the bidirectional long-short-term memory network model. The vital sign data can be detected in a non-contact and low-cost manner on the premise of ensuring that the vital sign data is accurately obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of vital sign data detection, and in particular to a vital sign data detection method, device, electronic equipment and storage medium. Background Art

[0002] With the progress of society, users are paying more and more attention to their own health. During the application process, users' vital signs data can be used to monitor their own health.

[0003] It is known from the related art that users can use wearable electronic devices or medical detection equipment to detect vital signs data. However, the detection of vital signs data based on wearable electronic devices often increases the user's usage cost; the detection of vital signs data based on medical detection equipment will require users to perform contact detection or invasive detection, causing psychological burden on users.

[0004] Therefore, finding a non-contact and cost-effective method to detect vital signs data has become a research hotspot. Summary of the invention

[0005] The present invention provides a vital sign data detection method, device, electronic equipment and storage medium, which realize the detection of vital sign data in a non-contact and low-cost manner.

[0006] The present invention provides a method for detecting vital sign data, the method comprising: acquiring multiple frames of continuous facial images of a user; obtaining multiple sets of feature data of preset regions of interest of the user's face based on the multiple frames of facial images; obtaining remote photoplethysmograms corresponding to the feature data based on the multiple sets of feature data; inputting the remote photoplethysmograms into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmograms; inputting the calibrated remote photoplethysmograms into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0007] According to a vital sign data detection method provided by the present invention, the calibration model is trained in the following manner: synchronously acquiring multiple frames of continuous facial image samples of a user and multiple groups of contact photoplethysmogram samples of the user, wherein the facial image samples and the contact photoplethysmogram samples have a corresponding relationship in the time dimension; based on the multiple frames of the facial image samples, obtaining multiple groups of feature data samples of a preset region of interest of the user's face; based on the multiple groups of the feature data samples, obtaining remote photoplethysmogram samples corresponding to the feature data samples; based on the remote photoplethysmogram samples and the contact photoplethysmogram samples that have a corresponding relationship with the remote photoplethysmogram samples in the time dimension, training the calibration model to obtain a trained calibration model.

[0008] According to a method for detecting vital sign data provided by the present invention, the feature data includes the channel value of any one of the RGB channels of the face image; the step of obtaining a remote photoplethysmogram corresponding to the feature data based on multiple groups of the feature data specifically includes: constructing a channel value curve formed by the channel values ​​and the time information based on the multiple groups of the channel values ​​and the time information corresponding to each of the channel values; and obtaining a remote photoplethysmogram corresponding to the feature data based on the channel value curve.

[0009] According to a vital sign data detection method provided by the present invention, the feature data sample includes a channel sample value of any one channel in the RGB channel samples of the face image sample; the remote photoplethysmogram sample corresponding to the feature data sample is obtained based on multiple groups of the feature data samples, specifically including: based on the multiple groups of the channel sample values ​​and the time information corresponding to each of the channel sample values, a channel sample value curve formed by the channel sample values ​​and the time information is constructed; based on the channel sample value curve, a remote photoplethysmogram sample corresponding to the feature data sample is obtained.

[0010] According to a method for detecting vital signs data provided by the present invention, the vital signs data includes multiple vital signs data; after obtaining the vital signs data output by the bidirectional long short-term memory network model, the method further includes: obtaining a comprehensive score of the vital signs data of the user's vital signs data based on the multiple vital signs data; determining a vital signs level corresponding to the comprehensive score of the vital signs data based on the comprehensive score of the vital signs data and a pre-constructed first mapping table, wherein the first mapping table includes a correspondence between different comprehensive scores of vital signs data and different vital signs levels; and providing a dietary recipe corresponding to the vital signs level for the user based on the vital signs level.

[0011] According to a vital signs data detection method provided by the present invention, the comprehensive score of vital signs data of the user's vital signs data is obtained based on the multiple vital signs data, specifically including: obtaining the vital signs data score of each vital signs data based on each vital signs data and the vital signs data standard value corresponding to each vital signs data; respectively determining the weight value of each vital signs data; and obtaining the comprehensive score of vital signs data of the user's vital signs data by weighted summation based on the vital signs data score of each vital signs data and the weight value of each vital signs data.

[0012] According to a method for detecting vital sign data provided by the present invention, based on the vital sign level, providing the user with a dietary recipe corresponding to the vital sign level, specifically includes: pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different vital sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes; based on the vital sign level, determining a dietary recipe set corresponding to the vital sign level; acquiring the taste preference information of the user; and based on the taste preference information, determining the dietary recipe in the dietary recipe set that matches the taste preference information.

[0013] According to a method for detecting vital sign data provided by the present invention, after providing a dietary recipe corresponding to the physical sign level to the user based on the physical sign level, the method further includes: determining a recipe calorie value corresponding to the dietary recipe based on the dietary recipe; determining a standard reserved calorie value corresponding to the physical sign level based on the physical sign level; determining a calorie value that the user should consume based on the standard reserved calorie value and the recipe calorie value; and determining an exercise plan that matches the physical sign level of the user and the dietary recipe based on the calorie value that should be consumed.

[0014] The present invention also provides a vital sign data detection device, which includes: an acquisition module, which is used to acquire multiple frames of continuous facial images of a user; a feature data determination module, which is used to obtain multiple groups of feature data of a preset area of ​​interest of the user's face based on multiple frames of the facial images; a remote photoplethysmogram determination module, which is used to obtain a remote photoplethysmogram corresponding to the feature data based on multiple groups of the feature data; a processing module, which is used to input the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; and a generation module, which is used to input the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a vital sign data detection method as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the vital sign data detection method as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for detecting vital sign data as described in any one of the above is implemented.

[0018] The vital sign data detection method, device, electronic device and storage medium provided by the present invention obtain multiple sets of feature data of preset interest areas of the user's face based on the continuous multi-frame face images of the user; obtain remote photoplethysmograms corresponding to the feature data based on the multiple sets of feature data; then input the remote photoplethysmograms into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, and input the calibrated remote photoplethysmograms into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmogram is obtained through calibration, it can be ensured that the calibrated remote photoplethysmogram is closer to the real remote photoplethysmogram, thereby ensuring that the vital sign data finally obtained is more accurate. Thus, it is achieved that the detection of vital sign data can be achieved in a non-contact, low-cost manner under the premise of ensuring that the vital sign data is accurately obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is one of the flow charts of the vital sign data detection method provided by the present invention;

[0021] Figure 2 It is a flow chart of the training calibration model provided by the present invention;

[0022] Figure 3 This is the second flow chart of the vital sign data detection method provided by the present invention;

[0023] Figure 4 This is the third flow chart of the vital sign data detection method provided by the present invention;

[0024] Figure 5 It is a structural schematic diagram of the vital sign data detection device provided by the present invention;

[0025] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In one embodiment, with the popularization of smart mobile terminal devices, such as mobile phones, tablets, smart home central control screens, laptops, smart magic mirrors, smart refrigerators, etc., and these smart mobile terminal devices are equipped with image acquisition devices, so that the vital signs data detection solution based on multi-frame images can be realized without increasing any hardware costs. Through the image acquisition device carried by the smart mobile terminal device itself, such as a camera, and using remote photoelectric volume pulse wave technology, non-contact vital signs data including heart rate, blood pressure, blood oxygen, blood sugar and other indicators can be realized in real-time and continuous detection.

[0028] The executor of the vital sign data detection method provided by the present invention can be a smart mobile terminal device, which has no special requirements for the hardware and system of the smart mobile terminal device, has no strict requirements for the camera hardware configuration and can be upward compatible. The method of obtaining the user's vital sign indicators can be an edge acquisition method or a cloud acquisition method. In another example, the vital sign data can be associated with the user's dietary health, exercise management, body management and other scenarios in real time, thereby providing users with active, imperceptible, customized and personalized solutions.

[0029] It should be noted that this solution has the advantages of low detection hardware cost, flexible, real-time and continuous detection, rich detection indicators, and association with user healthy diet and other active, imperceptible and personalized service scenarios.

[0030] Figure 1 This is one of the flow charts of the vital sign data detection method provided by the present invention.

[0031] In order to further introduce the vital sign data detection method provided by the present invention, Figure 1Provide explanation.

[0032] In an exemplary embodiment of the present invention, Figure 1 It can be seen that the vital sign data detection method may include steps 110 to 150, and each step will be introduced below.

[0033] In step 110, multiple frames of continuous facial images of the user are obtained.

[0034] In step 120, multiple groups of feature data of the user's preset area of ​​interest are obtained based on multiple frames of facial images.

[0035] In one embodiment, multiple consecutive frames of facial images of the user can be collected based on the inherent image acquisition device on the smart mobile terminal device, and multiple sets of feature data of preset regions of interest on the user's face can be obtained based on the multiple consecutive frames of facial images. The preset region of interest on the face is a preset region in the facial image, which can be adjusted according to actual conditions, such as the forehead, cheek, and mouth. In this embodiment, the preset region of interest on the face is not specifically limited. In this embodiment, multiple consecutive frames of facial images of the user are collected based on the inherent image acquisition device on the smart mobile terminal device, so that facial images can be obtained without increasing any hardware costs, and a foundation can be laid for obtaining vital sign data based on facial images without incurring any costs.

[0036] In another example, the feature data does not contain data information for identifying the user, so that the feature data cannot be restored to the user's face image and video, thereby avoiding the leakage of the user's privacy.

[0037] In step 130, based on the multiple sets of characteristic data, a remote photoplethysmogram corresponding to the characteristic data is obtained.

[0038] In another exemplary embodiment of the present invention, the feature data includes a channel value of any one of the RGB channels of the face image, for example, an R channel value, a G channel value, or a B channel value. Based on multiple sets of feature data, obtaining a remote photoplethysmogram corresponding to the feature data can be implemented in the following manner:

[0039] Based on multiple groups of channel values ​​and time information corresponding to each channel value, construct a channel value curve formed by the channel values ​​and the time information;

[0040] Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

[0041] In one embodiment, since the multiple groups of channel values ​​are channel values ​​under continuous frames, the time information corresponding to each channel value can be obtained. Furthermore, based on the multiple groups of channel values ​​and the time information corresponding to each channel value, a channel value curve formed by the channel value and the time information can be constructed. In other words, the channel value curve can be obtained with the channel value as the horizontal coordinate and the time information as the vertical coordinate. In the application process, the remote photoelectric volume pulse wave corresponding to the feature data can be obtained based on the channel value curve. Among them, the aforementioned channel values ​​can be determined based on the average value of the channel values ​​of each pixel point in the preset area of ​​interest of the face.

[0042] In step 140, the remote photoplethysmogram is input into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model.

[0043] In step 150, the calibrated remote photoplethysmogram is input into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0044] In one embodiment, the obtained remote photoplethysmogram can be input into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model. The remote photoplethysmogram is a photoplethysmogram obtained based on a non-contact method. The remote photoplethysmogram may be inaccurate. Therefore, in order to improve the accuracy of the remote photoplethysmogram obtained based on a non-contact method, the remote photoplethysmogram needs to be calibrated. The calibration model is a network model, for example, it can be a U-NET network model. The calibration model is used to calibrate the remote photoplethysmogram to obtain a calibrated remote photoplethysmogram. The calibration model is pre-trained based on a sample set.

[0045] During the application process, the remote photoplethysmogram can be calibrated based on a pre-trained calibration model to obtain a calibrated remote photoplethysmogram. Furthermore, the calibrated remote photoplethysmogram is input into a pre-trained bidirectional long short-term memory network model, so that the vital signs data output by the bidirectional long short-term memory network model can be obtained. Since the calibrated remote photoplethysmogram is obtained through calibration, it can be ensured that the calibrated remote photoplethysmogram is closer to the real remote photoplethysmogram, so that the final vital signs data can be more accurate. In this way, it is achieved that on the premise of ensuring that the vital signs data are accurately obtained, the detection of vital signs data can be achieved in a non-contact, low-cost manner.

[0046] In another example, the bidirectional long short-term memory network model can be trained based on the calibrated remote photoplethysmography sample set to obtain a trained bidirectional long short-term memory network model. In other words, the bidirectional long short-term memory network model is obtained by pre-training with the sample training set.

[0047] The vital sign data detection method provided by the present invention obtains multiple sets of feature data of the user's preset area of ​​interest based on the user's continuous multi-frame face images; obtains remote photoplethysmograms corresponding to the feature data based on the multiple sets of feature data; then inputs the remote photoplethysmograms into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, and inputs the calibrated remote photoplethysmograms into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmogram is obtained through calibration, it can be ensured that the calibrated remote photoplethysmogram is closer to the real remote photoplethysmogram, thereby ensuring that the ultimately obtained vital sign data is more accurate. This achieves that, on the premise of ensuring that the vital sign data is accurately obtained, the detection of vital sign data can be achieved in a non-contact, low-cost manner.

[0048] Figure 2 It is a flow chart of the training calibration model provided by the present invention.

[0049] The following will be combined Figure 2 Describe the process of training and calibrating the model.

[0050] In an exemplary embodiment of the present invention, Figure 2 It can be seen that training the calibration model may include steps 210 to 240, and each step will be introduced below.

[0051] In step 210, a plurality of consecutive frames of facial image samples of the user and a plurality of groups of contact photoplethysmogram samples of the user are synchronously acquired respectively.

[0052] In one embodiment, multiple consecutive frames of facial image samples of the user and multiple groups of contact photoplethysmogram samples of the user may be acquired synchronously and separately. In other words, there is a corresponding relationship between the multiple consecutive frames of facial image samples and the contact photoplethysmogram samples in the time dimension.

[0053] In yet another embodiment, the contact photoplethysmogram sample may be obtained based on a contact test between the detection device and the user.

[0054] In step 220, based on the multiple frames of facial image samples, multiple groups of feature data samples of the preset regions of interest of the user's face are obtained.

[0055] In step 230, based on the multiple groups of characteristic data samples, remote photoplethysmogram samples corresponding to the characteristic data samples are obtained.

[0056] In step 240, the calibration model is trained based on the remote photoplethysmography samples and the contact photoplethysmography samples that correspond to the remote photoplethysmography samples in the time dimension to obtain a trained calibration model.

[0057] In one embodiment, multiple groups of feature data samples of a preset region of interest of the user's face can be obtained based on multiple frames of face image samples. Since there is a corresponding relationship between the continuous multiple frames of face image samples and the contact photoplethysmography samples in the time dimension, and the multiple groups of feature data samples are obtained based on the continuous multiple frames of face image samples, there is also a corresponding relationship between the multiple groups of feature data samples and the contact photoplethysmography samples in time.

[0058] Furthermore, the calibration model can be trained based on remote photoplethysmogram samples and contact photoplethysmogram samples that correspond to the remote photoplethysmogram samples in the time dimension to obtain a trained calibration model. Since the contact photoplethysmogram samples are obtained based on the contact test between the detection device and the user, the contact photoplethysmogram samples are more accurate. In the present application, the remote photoplethysmogram samples can be calibrated using the contact photoplethysmogram samples as the true value. In the present application, the calibration model is trained based on the above ideas to ensure that the remote photoplethysmogram input into the trained calibration model can obtain a more accurate calibrated remote photoplethysmogram.

[0059] It should be noted that the physical meaning of face image samples and face images is the same, the difference is that face image samples are training samples, while face images are face images to be processed. Based on the same principle, the relationship between contact photoplethysmography samples and contact photoplethysmography, feature data samples and feature data, remote photoplethysmography samples and remote photoplethysmography is also like the relationship between face image samples and face images.

[0060] In another exemplary embodiment of the present invention, the feature data sample may include a channel sample value of any channel in the RGB channel samples of the face image sample. Further, based on multiple groups of feature data samples, obtaining a remote photoplethysmogram sample corresponding to the feature data sample (corresponding to step 230) may be implemented in the following manner:

[0061] Based on multiple groups of channel sample values ​​and time information corresponding to each channel sample value, construct a channel sample value curve formed by the channel sample values ​​and the time information;

[0062] Based on the channel sample value curve, a remote photoplethysmogram sample corresponding to the characteristic data sample is obtained.

[0063] The principle of obtaining the remote photoplethysmogram sample corresponding to the characteristic data sample is the same as or similar to the principle of obtaining the remote photoplethysmogram corresponding to the characteristic data, and will not be described in detail in this embodiment.

[0064] Figure 3 This is the second flow chart of the vital sign data detection method provided by the present invention.

[0065] In order to further introduce the vital sign data detection method provided by the present invention, Figure 3 Provide explanation.

[0066] In an exemplary embodiment of the present invention, the vital sign data may include multiple vital sign data, such as heart rate, blood pressure, blood oxygen (PO2), standard deviation of sinus beat RR interval (SDNN), blood sugar, respiratory rate, pressure value, glycosylated hemoglobin (A1C), etc. In this embodiment, the vital sign data is not specifically limited.

[0067] Combination Figure 3 It can be seen that after obtaining the vital sign data output by the bidirectional long short-term memory network model (corresponding to step 150), the vital sign data detection method can also include steps 310 to 330, and each step will be introduced below.

[0068] In step 310 , a comprehensive score of the vital sign data of the user's vital sign data is obtained based on the plurality of vital sign data.

[0069] In one embodiment, the vital sign data detection method can also be implemented based on an APP installed on a smart mobile terminal device. Furthermore, multiple vital sign data can be obtained, and a comprehensive score of the vital sign data of the user's vital sign data can be obtained based on the vital sign data. It can be understood that the comprehensive score of the vital sign data is a score obtained by comprehensively considering each vital sign, so as to ensure that the comprehensive score of the vital sign data obtained is more accurate and reasonably matches the user's vital signs, thereby laying a foundation for providing users with reasonable dietary recipes.

[0070] In step 320, the physical sign level corresponding to the physical sign data comprehensive score is determined based on the physical sign data comprehensive score and a pre-constructed first mapping table, wherein the first mapping table includes a correspondence between different physical sign data comprehensive scores and different physical sign levels.

[0071] In step 330 , based on the physical sign level, a dietary recipe corresponding to the physical sign level is provided to the user.

[0072] In one embodiment, a first mapping table may be pre-constructed, wherein the first mapping table includes a correspondence between different comprehensive scores of vital sign data and different vital sign levels. In one example, comprehensive score A of vital sign data corresponds to vital sign level a; comprehensive score B of vital sign data corresponds to vital sign level b. During the application process, when it is determined that the comprehensive score of the user's vital sign data is comprehensive score A of vital sign data, the user's vital sign level may be determined to be vital sign level a. Furthermore, based on the vital sign level, a dietary recipe corresponding to the vital sign level may be provided to the user, wherein the dietary recipe may include the ingredients to be ingested and the intake of the corresponding ingredients. Through this embodiment, reasonable dietary recipes may be proactively provided to the user based on the obtained vital sign data.

[0073] In another exemplary embodiment of the present invention, based on multiple vital sign data, obtaining a comprehensive score of the vital sign data of the user can be implemented in the following manner:

[0074] Based on each vital sign data and the vital sign data standard value corresponding to each vital sign data, a vital sign data score of each vital sign data is obtained;

[0075] Determine the weight value of each vital sign data respectively;

[0076] A weighted sum is performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive vital sign data score of the user's vital sign data.

[0077] In one embodiment, the standard value of vital sign data corresponding to each vital sign data may be obtained respectively, and based on the vital sign data and the standard value of vital sign data corresponding to the vital sign data, the vital sign data score of the vital sign data may be obtained.

[0078] Now, let's take blood pressure as an example. If the user's blood pressure is m, the standard value of blood pressure is M. In the application process, the blood pressure vital sign data score can be determined based on the difference between the standard value M of blood pressure and blood pressure m. It can be understood that the difference between the standard value M of blood pressure and blood pressure m is negatively correlated with the blood pressure vital sign data score.

[0079] Furthermore, the weight values ​​of each vital sign data can be obtained. The weight values ​​can be determined according to the importance of the vital sign data. The higher the importance of the vital sign data, the greater the corresponding weight value. In one example, the vital sign data are blood pressure and weight. Since the importance of blood pressure is greater than that of weight, the weight value corresponding to blood pressure is greater than the weight value corresponding to weight.

[0080] In another embodiment, a weighted sum can be performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive score of the vital sign data of the user. Continuing to use blood pressure and weight as an example for the vital sign data, the weighted sum can be performed on the respective vital sign data scores and weight values ​​of blood pressure and weight, so as to obtain a comprehensive score of the vital sign data of the user. This ensures that the obtained comprehensive score of vital sign data comprehensively considers each vital sign data and the importance of each vital sign data, laying a foundation for recommending a reasonable diet recipe based on the comprehensive score of vital sign data.

[0081] In another exemplary embodiment of the present invention, based on the physical sign level, providing the user with a dietary recipe corresponding to the physical sign level can be implemented in the following manner:

[0082] Pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes;

[0083] Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level;

[0084] Obtain user's taste preference information;

[0085] Based on the taste preference information, a meal recipe matching the taste preference information is determined in the meal recipe collection.

[0086] In one embodiment, a second mapping table may be pre-constructed, wherein the second mapping table may include correspondences between different physical sign levels and different dietary recipe sets. In one example, physical sign level a may correspond to dietary recipe set a'; physical sign level b may correspond to dietary recipe set b'.

[0087] Further, when it is determined that the physical sign level of the user is physical sign level a, the dietary recipe set corresponding to the user may be determined to be a dietary recipe set a', wherein the dietary recipe set a' includes a plurality of dietary recipes.

[0088] In another embodiment, the user's taste preference information can also be obtained, and further, based on the user's taste preference information, a dietary recipe matching the taste preference information is determined in the dietary recipe set. During the application process, the determined dietary recipe matching the taste preference information can be used as a dietary recipe recommended for the user.

[0089] In another embodiment, the dietary recipes matching the taste preference information may be used as dietary recipes recommended to the user. Through this embodiment, it can be ensured that the dietary recipes recommended to the user not only meet the user's physical sign level, but also meet the user's taste preferences, thereby improving the user's experience and satisfaction.

[0090] Figure 4 This is the third flow chart of the vital sign data detection method provided by the present invention.

[0091] In another exemplary embodiment of the present invention, Figure 4 It can be seen that after providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level, the vital sign data detection method can also include steps 410 to 440, and each step will be introduced below.

[0092] In step 410 , based on the dietary recipe, a recipe calorie value corresponding to the dietary recipe is determined.

[0093] In step 420, based on the physical sign level, a standard conservation calorie value corresponding to the physical sign level is determined.

[0094] In one embodiment, the calorie value of the recipe corresponding to the dietary recipe recommended for the user can be determined. Further, based on the physical sign level, the standard preservation calorie value corresponding to the physical sign level is determined. It can be understood that different physical sign levels can correspond to different standard preservation calorie values. When the user's physical sign level is higher, the corresponding standard consumption calorie value can be higher. Among them, the standard preservation calorie value is the calorie value that the user maintains the user's reasonable physical sign needs.

[0095] In step 430, the calorie value that the user should consume is determined based on the standard saved calorie value and the recipe calorie value.

[0096] In step 440 , an exercise program matching the user's physical sign level and dietary recipe is determined based on the calorie value to be consumed.

[0097] In one embodiment, the calorie value that the user should consume can be determined based on the difference between the calorie value of the recipe and the standard saved calorie value. Further, based on the calorie value that should be consumed, an exercise plan that matches the user's physical sign level and dietary recipe can be determined, and the exercise plan can be actively provided to the user. In this way, it can be ensured that the user's body management is ensured through the exercise plan under the premise of meeting the user's physical sign level and matching the user's dietary recipe.

[0098] According to the above description, the vital signs data detection method provided by the present invention obtains multiple sets of feature data of the user's preset area of ​​interest based on the user's continuous multi-frame face images; obtains remote photoplethysmograms corresponding to the feature data based on the multiple sets of feature data; then inputs the remote photoplethysmograms into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, and inputs the calibrated remote photoplethysmograms into a pre-trained bidirectional long short-term memory network model to obtain vital signs data output by the bidirectional long short-term memory network model. Since the calibrated remote photoplethysmogram is obtained through calibration, it can be ensured that the calibrated remote photoplethysmogram is closer to the real remote photoplethysmogram, thereby ensuring that the vital signs data finally obtained is more accurate. In addition, it is achieved that the detection of vital signs data can be achieved in a non-contact, low-cost manner under the premise of ensuring that the vital signs data are accurately obtained.

[0099] Based on the same concept, the present invention also provides a vital sign data detection device.

[0100] The vital sign data detection device provided by the present invention is described below. The vital sign data detection device described below and the vital sign data detection method described above can be referred to each other.

[0101] Figure 5 It is a structural schematic diagram of the vital sign data detection device provided by the present invention.

[0102] In an exemplary embodiment of the present invention, Figure 5 It can be seen that the vital sign data detection device can include an acquisition module 510, a feature data determination module 520, a remote photoplethysmography determination module 530, a processing module 540 and a generation module 550, each of which will be introduced below.

[0103] The acquisition module 510 may be configured to acquire multiple consecutive frames of facial images of the user;

[0104] The feature data determination module 520 may be configured to obtain multiple sets of feature data of a preset region of interest of a user's face based on multiple frames of face images;

[0105] The remote photoplethysmography determination module 530 may be configured to obtain a remote photoplethysmography corresponding to the characteristic data based on the plurality of sets of characteristic data;

[0106] The processing module 540 may be configured to input the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram;

[0107] The generation module 550 may be configured to input the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

[0108] In an exemplary embodiment of the present invention, the processing module 540 may train the calibration model in the following manner:

[0109] synchronously acquiring multiple frames of continuous face image samples of the user and multiple groups of contact photoplethysmogram samples of the user, wherein the face image samples and the contact photoplethysmogram samples have a corresponding relationship in the time dimension;

[0110] Based on multiple frames of face image samples, multiple groups of feature data samples of preset areas of interest of the user's face are obtained;

[0111] Based on the multiple groups of characteristic data samples, remote photoplethysmogram samples corresponding to the characteristic data samples are obtained;

[0112] Based on the remote photoplethysmogram samples and the contact photoplethysmogram samples corresponding to the remote photoplethysmogram samples in the time dimension, the calibration model is trained to obtain a trained calibration model.

[0113] In an exemplary embodiment of the present invention, the feature data may include a channel value of any one of the RGB channels of the face image;

[0114] The remote photoplethysmography module 530 may obtain the remote photoplethysmography corresponding to the characteristic data based on multiple sets of characteristic data in the following manner:

[0115] Based on multiple groups of channel values ​​and time information corresponding to each channel value, construct a channel value curve formed by the channel values ​​and the time information;

[0116] Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

[0117] In an exemplary embodiment of the present invention, the feature data sample may include a channel sample value of any channel among the RGB channel samples of the face image sample;

[0118] The processing module 540 can obtain remote photoplethysmogram samples corresponding to the characteristic data samples based on multiple groups of characteristic data samples in the following manner:

[0119] Based on multiple groups of channel sample values ​​and time information corresponding to each channel sample value, construct a channel sample value curve formed by the channel sample values ​​and the time information;

[0120] Based on the channel sample value curve, a remote photoplethysmogram sample corresponding to the characteristic data sample is obtained.

[0121] In an exemplary embodiment of the present invention, the vital sign data may include a plurality of vital sign data;

[0122] The generation module 550 may also be configured to:

[0123] Based on the multiple vital sign data, a comprehensive score of the vital sign data of the user is obtained;

[0124] Based on the comprehensive score of the physical sign data and a pre-constructed first mapping table, determining the physical sign level corresponding to the comprehensive score of the physical sign data, wherein the first mapping table includes a correspondence between different comprehensive scores of the physical sign data and different physical sign levels;

[0125] Based on the physical sign level, a dietary recipe corresponding to the physical sign level is provided to the user.

[0126] In an exemplary embodiment of the present invention, the generation module 550 may also use the following method to obtain the comprehensive score of the vital sign data of the user based on multiple vital sign data:

[0127] Based on each vital sign data and the vital sign data standard value corresponding to each vital sign data, a vital sign data score of each vital sign data is obtained;

[0128] Determine the weight value of each vital sign data respectively;

[0129] A weighted sum is performed based on the vital sign data score of each vital sign data and the weight value of each vital sign data to obtain a comprehensive vital sign data score of the user's vital sign data.

[0130] In an exemplary embodiment of the present invention, the generation module 550 may also provide the user with a dietary recipe corresponding to the physical sign level based on the physical sign level in the following manner:

[0131] Pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes;

[0132] Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level;

[0133] Obtain user's taste preference information;

[0134] Based on the taste preference information, a meal recipe matching the taste preference information is determined in the meal recipe collection.

[0135] In an exemplary embodiment of the present invention, the generating module 550 may also be configured to:

[0136] Based on the dietary recipe, determining a recipe calorie value corresponding to the dietary recipe;

[0137] Based on the physical sign level, a standard conservation calorie value corresponding to the physical sign level is determined;

[0138] Determine the calorie value that the user should consume based on the standard saved calorie value and the recipe calorie value;

[0139] Based on the calorie value that should be consumed, an exercise plan that matches the user's physical sign level and dietary recipe is determined.

[0140] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the vital sign data detection method, which includes: obtaining a plurality of consecutive frames of face images of the user; obtaining a plurality of sets of feature data of a preset region of interest of the face of the user based on the plurality of frames of face images; obtaining a remote photoelectric volumetric pulse wave corresponding to the feature data based on the plurality of sets of feature data; inputting the remote photoelectric volumetric pulse wave into a pre-trained calibration model to obtain a calibrated remote photoelectric volumetric pulse wave output by the calibration model, wherein the calibration model is used to calibrate the remote photoelectric volumetric pulse wave; inputting the calibrated remote photoelectric volumetric pulse wave into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0141] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vital sign data detection method provided by the above methods, and the method includes: acquiring multiple frames of continuous facial images of a user; based on the multiple frames of facial images, obtaining multiple sets of feature data of a preset area of ​​interest of the user's face; based on the multiple sets of feature data, obtaining a remote photoplethysmogram corresponding to the feature data; inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; inputting the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0143] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vital sign data detection method provided by the above-mentioned methods, the method comprising: acquiring a plurality of continuous frames of facial images of a user; obtaining a plurality of sets of feature data of a preset region of interest of the user's face based on the plurality of frames of facial images; obtaining a remote photoplethysmogram corresponding to the feature data based on the plurality of sets of feature data; inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; inputting the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain the vital sign data output by the bidirectional long short-term memory network model.

[0144] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0146] It is further understood that, although the operations are described in a specific order in the accompanying drawings in the embodiments of the present invention, it should not be understood as requiring the operations to be performed in the specific order or serial order shown, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting vital signs data, characterized in that: The method comprises: Obtain multiple consecutive frames of facial images of the user; Based on the multiple frames of the facial images, obtaining multiple sets of feature data of a preset region of interest of the user's face; Based on the plurality of sets of characteristic data, obtaining remote photoplethysmograms corresponding to the characteristic data; Inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used to calibrate the remote photoplethysmogram; The calibrated remote photoplethysmogram is input into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

2. The method for detecting vital signs data according to claim 1, characterized in that: The calibration model is trained in the following way: Synchronously acquiring multiple consecutive frames of face image samples of the user and multiple groups of contact photoplethysmogram samples of the user, wherein the face image samples and the contact photoplethysmogram samples have a corresponding relationship in the time dimension; Based on the multiple frames of face image samples, multiple groups of feature data samples of the preset regions of interest of the user's face are obtained; Based on the plurality of sets of characteristic data samples, obtaining remote photoplethysmogram samples corresponding to the characteristic data samples; Based on the remote photoplethysmography samples and the contact photoplethysmography samples corresponding to the remote photoplethysmography samples in the time dimension, the calibration model is trained to obtain a trained calibration model.

3. The method for detecting vital signs data according to claim 1, characterized in that: The feature data includes a channel value of any one of the RGB channels of the face image; The obtaining, based on the plurality of sets of characteristic data, a remote photoplethysmogram corresponding to the characteristic data, specifically comprises: Based on the plurality of groups of channel values ​​and the time information corresponding to each of the channel values, constructing a channel value curve formed by the channel values ​​and the time information; Based on the channel value curve, a remote photoplethysmogram corresponding to the characteristic data is obtained.

4. The method for detecting vital signs data according to claim 2, characterized in that: The feature data sample includes a channel sample value of any channel in the RGB channel samples of the face image sample; The obtaining, based on the plurality of groups of characteristic data samples, remote photoplethysmogram samples corresponding to the characteristic data samples specifically includes: Based on the multiple groups of channel sample values ​​and the time information corresponding to each of the channel sample values, construct a channel sample value curve formed by the channel sample values ​​and the time information; Based on the channel sample value curve, a remote photoplethysmogram sample corresponding to the characteristic data sample is obtained.

5. The method for detecting vital signs data according to claim 1, characterized in that: The vital sign data includes a plurality of vital sign data; After obtaining the vital sign data output by the bidirectional long short-term memory network model, the method further includes: Based on the plurality of vital sign data, obtaining a comprehensive score of the vital sign data of the user; Based on the comprehensive score of the physical sign data and a pre-constructed first mapping table, determining the physical sign level corresponding to the comprehensive score of the physical sign data, wherein the first mapping table includes a correspondence between different comprehensive scores of the physical sign data and different physical sign levels; Based on the physical sign level, a dietary recipe corresponding to the physical sign level is provided to the user.

6. The method for detecting vital signs data according to claim 5, characterized in that: The obtaining of a comprehensive score of the vital sign data of the user's vital sign data based on the plurality of vital sign data specifically includes: Based on each of the vital sign data and the vital sign data standard value corresponding to each of the vital sign data, obtaining a vital sign data score for each of the vital sign data; Determining the weight value of each of the vital sign data respectively; A weighted sum is performed based on the physical sign data score of each of the vital sign data and the weight value of each of the vital sign data to obtain a physical sign data comprehensive score of the vital sign data of the user.

7. The method for detecting vital signs data according to claim 5, characterized in that: The providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level specifically includes: Pre-constructing a second mapping table, wherein the second mapping table includes a correspondence between different physical sign levels and different dietary recipe sets, wherein the dietary recipe set includes a plurality of dietary recipes; Based on the physical sign level, determining a set of dietary recipes corresponding to the physical sign level; Acquiring the user's taste preference information; Based on the taste preference information, the meal recipe matching the taste preference information is determined in the meal recipe set.

8. The method for detecting vital signs data according to claim 5 or 7, characterized in that: After providing the user with a dietary recipe corresponding to the physical sign level based on the physical sign level, the method further includes: Based on the dietary recipe, determining a recipe calorie value corresponding to the dietary recipe; Based on the physical sign level, determining a standard preservation calorie value corresponding to the physical sign level; Determining the calorie value that the user should consume based on the standard saved calorie value and the recipe calorie value; Based on the calorie value to be consumed, an exercise plan matching the physical sign level and the dietary recipe of the user is determined.

9. A vital sign data detection device, characterized in that: The device comprises: An acquisition module is used to acquire multiple consecutive frames of facial images of the user; A feature data determination module is used to obtain multiple sets of feature data of a preset region of interest of the user's face based on multiple frames of the face image; Determine a remote photoplethysmography module, for obtaining a remote photoplethysmography corresponding to the characteristic data based on a plurality of sets of the characteristic data; A processing module, used for inputting the remote photoplethysmogram into a pre-trained calibration model to obtain a calibrated remote photoplethysmogram output by the calibration model, wherein the calibration model is used for calibrating the remote photoplethysmogram; A generation module is used to input the calibrated remote photoplethysmogram into a pre-trained bidirectional long short-term memory network model to obtain vital sign data output by the bidirectional long short-term memory network model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the vital sign data detection method as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vital sign data detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vital sign data detection method according to any one of claims 1 to 8 is implemented.

Images