First-aid online vital sign monitoring method based on rPPG

Abstract

The invention provides a first-aid online vital sign monitoring method based on rPPG, and relates to the technical field of first aid. The method comprises the following steps: acquiring a face video of a to-be-detected face; for the face presented in the face video, a change image of subcutaneous blood volume is determined, and the change image comprises a pre-change image and a post-change image; the rPPG is adopted to process the change image so as to judge whether the vital signs corresponding to the face are preset signs to be subjected to emergency treatment or not; and if the vital sign is the preset sign to be subjected to emergency treatment, generating and outputting a message to be subjected to emergency treatment. According to the method, whether the vital signs corresponding to the face are the signs to be subjected to emergency treatment or not can be judged through rPPG detection, then whether emergency treatment is needed or not is judged, and a judgment basis is provided for pre-hospital emergency treatment.

Description

Technical Field

[0001] The present disclosure relates to the field of emergency medical technology, and in particular to an online vital sign monitoring method based on rPPG emergency medical care. Background Art

[0002] rPPG (Remote Photo-plethysmography) is a technology that uses a camera to obtain periodic subtle skin color changes caused by the cardiac cycle that are invisible to the naked eye, thereby extracting pulse waves. This technology has the advantages of being non-sensitive, fast and low-cost.

[0003] Before emergency treatment, it is often necessary to quickly determine the patient's vital signs to determine whether the patient needs pre-hospital emergency treatment. Therefore, how to apply rPPG technology to emergency treatment is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to propose an online vital signs monitoring method based on rPPG emergency treatment, which can solve the existing problems in a targeted manner.

[0005] Based on the above-mentioned purpose, in the first aspect, the present disclosure proposes an online vital signs monitoring method based on rPPG first aid, comprising: obtaining a facial video of a face to be detected; for the face presented in the facial video, determining a change image of subcutaneous blood volume, wherein the change image comprises an image before the change and an image after the change; using the rPPG to process the change image to determine whether the vital signs corresponding to the face are preset signs to be urgently needed; if the vital signs are preset signs to be urgently needed, generating and outputting a message to be urgently needed.

[0006] In the second aspect, an online vital sign monitoring device based on rPPG first aid is also provided, including: an acquisition unit, configured to acquire a facial video of a face to be detected; a determination unit, configured to determine a change image of subcutaneous blood volume for the face presented in the facial video, the change image including an image before the change and an image after the change; a judgment unit, configured to use the rPPG to process the change image to determine whether the vital sign corresponding to the face is a preset sign to be urgently needed; a generation unit, configured to generate and output a message to be urgently needed if the vital sign is a preset sign to be urgently needed.

[0007] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement any method described in the first aspect.

[0009] In a fifth aspect, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to implement any method described in the first aspect.

[0010] In general, the present disclosure has at least the following beneficial effects: rPPG detection can be used to determine whether the vital signs corresponding to the face are signs requiring emergency treatment, and then determine whether emergency treatment is required, providing a basis for judgment for pre-hospital emergency treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0012] Figure 1a A flow chart of a method for online vital sign monitoring based on rPPG first aid according to an embodiment of the present disclosure is shown; Figure 1b A schematic diagram of a multi-spectral remote photoplethysmography rPPG according to an embodiment of the present disclosure is shown; Figure 1c shows an oxygenated hemoglobin distribution image according to an embodiment of the present disclosure; Figure 1d shows a blood flow pulsation distribution image according to an embodiment of the present disclosure; Figure 1e shows a hemoglobin distribution image according to an embodiment of the present disclosure; Figure 2 A schematic diagram of a vital sign detection device for emergency treatment based on multi-spectral remote photoplethysmography rPPG according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown; Figure 4 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0013] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0014] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0015] Figure 1a The present invention shows a method for online vital sign monitoring based on rPPG emergency treatment. In an embodiment of the present invention, the method comprises: Step S101, obtaining a facial video of a face to be detected.

[0016] Step S102, determining a change image of subcutaneous blood volume for the face presented in the facial video, wherein the change image includes an image before the change and an image after the change.

[0017] Step S103: using the rPPG to process the changed image to determine whether the vital signs corresponding to the face are preset signs to be rescued.

[0018] Step S104: if the vital sign is a preset emergency sign, a emergency message is generated and outputted.

[0019] In this embodiment, the execution subject of the rPPG-based emergency online vital signs monitoring method can obtain a facial video, determine a changed image from the image in the facial video, and then use rPPG to process the changed image to determine whether to output a waiting emergency message.

[0020] like Figure 1b As shown, the figure shows the schematic diagram of rPPG.

[0021] The execution subject may use a preset model to generate a change image from an image of the face presented in a facial video. The change image may reflect the subcutaneous blood volume, such as a blood flow pulsation distribution image. The image before the change corresponds to a video image in the facial video, and the image after the change corresponds to another video image. There is an interval between the two video images.

[0022] This embodiment can determine whether the vital signs corresponding to the face are signs for emergency treatment through rPPG detection, and then determine whether emergency treatment is needed, providing a judgment basis for pre-hospital emergency treatment.

[0023] In some optional implementations of any one of the embodiments of the present disclosure, the using the rPPG to process the changed image to determine whether the vital signs corresponding to the face are preset signs to be treated with emergency treatment, includes: obtaining the interval duration between the image before the change and the image after the change; using the rPPG model to process the interval duration and the changed image to obtain emergency treatment indication information, and determining whether the vital signs corresponding to the emergency treatment indication information are preset signs to be treated with emergency treatment, wherein the emergency treatment indication information is used to indicate the probability of the face corresponding to the need for emergency treatment.

[0024] In these implementations, the rPPG may be an rPPG model. The model may be a deep neural network capable of image processing, such as a convolutional neural network. The model may be trained in advance using the changing images and the interval duration, so that the model can predict the emergency indication information through the changing images and the interval duration.

[0025] In some application scenarios of these optional implementations, the change image includes the following images: a facial image, an oxygenated hemoglobin O2Hb distribution image, a blood flow pulsation distribution image, and a hemoglobin HHb distribution image; the rPPG model is used to process the interval duration and the change image to obtain emergency response indication information, including: determining the vector product of the image before the change and the image after the change for each change image; inputting the interval duration, the facial image, and the vector product corresponding to each change image into the rPPG model to obtain first indication information corresponding to the facial image, second indication information corresponding to the oxygenated hemoglobin distribution image, third indication information corresponding to the blood flow pulsation distribution image, and fourth indication information corresponding to the hemoglobin distribution image, and the emergency response indication information includes the first indication information, the second indication information, the third indication information, and the fourth indication information.

[0026] The oxygenated hemoglobin distribution image is an image indicating the distribution position of oxygenated hemoglobin under the facial skin. The blood flow pulsation distribution image is an image indicating the distribution position of blood flow pulsation under the facial skin. The hemoglobin distribution image is an image indicating the distribution position of deoxyhemoglobin under the facial skin.

[0027] Optionally, the first indication information is used to indicate the probability of needing first aid corresponding to the facial image, the second indication information is used to indicate the probability of needing first aid corresponding to the oxygenated hemoglobin distribution image, the third indication information is used to indicate the probability of needing first aid corresponding to the blood flow pulsation distribution image, and the fourth indication information is used to indicate the probability of needing first aid corresponding to the hemoglobin distribution image.

[0028] like Figure 1c As shown in FIG. 1 , the oxygenated hemoglobin distribution image is shown. Figure 1dAs shown in FIG. 1 , the blood flow pulsation distribution image is shown. Figure 1e As shown, a hemoglobin distribution image is shown in the figure.

[0029] The first indication information, the second indication information, the third indication information, the fourth indication information and the first aid indication information here may be numerical information.

[0030] In some cases of these application scenarios, the first indication information, the second indication information, the third indication information and the fourth indication information are all numerical values; the judging whether the vital signs corresponding to the indication information to be first aid are preset vital signs to be first aid include: determining the exponential function values ​​of the standard deviations for the first indication information, the second indication information, the third indication information and the fourth indication information, respectively, to obtain the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value; determining the weights for the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value, respectively, in real time; weighting the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value using the respective weights to obtain the indication information to be first aid; determining the vital signs corresponding to the indication information to be first aid, and judging whether the vital signs are preset vital signs to be first aid.

[0031] In these cases, the determined weights change dynamically with different facial states. The weights may be determined in various ways, such as inputting the changed image into a preset weight determination model to determine the weight corresponding to the changed image.

[0032] There may be a preset mapping relationship between the emergency indication information and the vital signs, and the vital signs can be determined using the mapping relationship. Alternatively, a preset vital sign determination model can be used to determine the vital signs from the emergency indication information.

[0033] Optionally, the exponential function value is calculated using an exponential function, and the exponential function is expressed as: X 10*X , where X is the first indication information, the second indication information, the third indication information or the fourth indication information.

[0034] Optionally, the real-time weighting of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value are respectively determined, including: determining the subcutaneous blood volume of the image with the last playback time in the facial video; and determining the weights of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value according to the subcutaneous blood volume.

[0035] The execution subject may determine the weights of the first exponential function value, the second exponential function value, the third exponential function value, and the fourth exponential function value according to the subcutaneous blood volume in various ways. For example, through the correspondence between the subcutaneous blood volume, the exponential function, and the weight, the weight corresponding to the subcutaneous blood volume and the first exponential function, the weight corresponding to the subcutaneous blood volume and the second exponential function, the weight corresponding to the subcutaneous blood volume and the third exponential function, and the weight corresponding to the subcutaneous blood volume and the fourth exponential function may be determined. Alternatively, the subcutaneous blood volume and the exponential function may be input into a weight determination formula or model to determine the weight.

[0036] Specifically, different subcutaneous blood volume ranges correspond to different weights of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value; determining the weights of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value according to the subcutaneous blood volume includes: determining the subcutaneous blood volume range where the subcutaneous blood volume is located, and determining weights for the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value respectively according to the subcutaneous blood volume range.

[0037] Optionally, determining the vital signs corresponding to the first aid indication information and judging whether the vital signs are preset first aid signs include: determining the vital signs values ​​corresponding to the face according to the first aid indication information, the vital signs values ​​including the heart rate value and the blood pressure value; judging whether the vital signs values ​​are within a preset range of first aid signs; if the vital signs values ​​are within the preset range of first aid signs, then the vital signs are preset first aid signs.

[0038] The disclosed embodiment provides a device for detecting vital signs based on multi-spectral remote photoplethysmography rPPG emergency treatment, which is used to perform the method for online vital signs monitoring based on rPPG emergency treatment described in the above embodiment, such as Figure 2 As shown, the device comprises: The acquisition unit 201 is configured to acquire a facial video of a face to be detected; the determination unit 202 is configured to determine a change image of subcutaneous blood volume for the face presented in the facial video, wherein the change image includes an image before the change and an image after the change; the judgment unit 203 is configured to process the change image using the rPPG to determine whether the vital signs corresponding to the face are preset signs to be urgently needed; the generation unit 204 is configured to generate and output a message to be urgently needed if the vital signs are preset signs to be urgently needed.

[0039] The vital signs detection device based on multi-spectral remote photoplethysmography rPPG first aid provided in the above-mentioned embodiments of the present disclosure and the online vital signs monitoring method based on rPPG first aid provided in the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0040] The embodiment of the present disclosure also provides an electronic device corresponding to the rPPG-based online emergency vital sign monitoring method provided in the above embodiment, so as to execute the above rPPG-based online emergency vital sign monitoring method. The embodiment of the present disclosure is not limited.

[0041] Please refer to Figure 3 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 3 As shown, the electronic device 30 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, and the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and the processor 300 executes the method provided in any of the aforementioned embodiments of the present disclosure when running the computer program.

[0042] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0043] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving the execution instructions. The rPPG-based emergency online vital sign monitoring method disclosed in any of the embodiments of the present disclosure may be applied to the processor 300, or implemented by the processor 300.

[0044] The processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 300. The above processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0045] The electronic device provided in the embodiment of the present disclosure and the rPPG-based emergency online vital sign monitoring method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0046] The present disclosure also provides a computer-readable storage medium corresponding to the rPPG-based online emergency vital sign monitoring method provided in the above embodiment. Figure 4 The computer-readable storage medium shown is a CD 40 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the rPPG-based emergency online vital sign monitoring method provided in any of the aforementioned embodiments will be executed.

[0047] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0048] The computer-readable storage medium provided in the above-mentioned embodiments of the present disclosure and the rPPG-based emergency online vital signs monitoring method provided in the embodiments of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0049] It should be noted that: In the above text, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0051] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the purpose of the present disclosure and the scope of protection of the claims, all of which are within the protection of the present disclosure.

Claims

1. A method for online vital sign monitoring based on rPPG emergency treatment, characterized in that: The method comprises: Get the facial video of the face to be detected; For the face presented in the facial video, determining a change image of subcutaneous blood volume, wherein the change image includes an image before the change and an image after the change; Processing the changed image using the rPPG to determine whether the vital signs corresponding to the face are preset emergency signs; If the vital sign is a preset emergency sign, a first aid message is generated and output.

2. The method according to claim 1, characterized in that The using the rPPG to process the changed image to determine whether the vital signs corresponding to the face are preset emergency signs includes: Acquire the interval length between the image before the change and the image after the change; The rPPG model is used to process the interval duration and the change image to obtain first aid indication information, and it is determined whether the vital signs corresponding to the first aid indication information are preset first aid signs. The first aid indication information is used to indicate the probability of first aid corresponding to the face.

3. The method according to claim 2, characterized in that The change images include the following images: facial image, oxygenated hemoglobin distribution image, blood flow pulsation distribution image and hemoglobin distribution image; The rPPG model is used to process the interval duration and the change image to obtain the emergency instruction information, including: For each changed image, determining a vector product of the image before the change and the image after the change; The interval duration, the facial image, and the vector product corresponding to each change image are input into the rPPG model to obtain first indication information corresponding to the facial image, second indication information corresponding to the oxygenated hemoglobin distribution image, third indication information corresponding to the blood flow pulsation distribution image, and fourth indication information corresponding to the hemoglobin distribution image. The first aid indication information includes the first indication information, the second indication information, the third indication information, and the fourth indication information.

4. The method according to claim 3, characterized in that The first indication information is used to indicate the probability of needing first aid corresponding to the facial image, the second indication information is used to indicate the probability of needing first aid corresponding to the oxygenated hemoglobin distribution image, the third indication information is used to indicate the probability of needing first aid corresponding to the blood flow pulsation distribution image, and the fourth indication information is used to indicate the probability of needing first aid corresponding to the hemoglobin distribution image.

5. The method according to claim 3, characterized in that: The first indication information, the second indication information, the third indication information and the fourth indication information are all numerical values; The determining whether the vital sign corresponding to the first aid indication information is a preset first aid sign includes: Determine exponential function values ​​of standard deviations for the first indication information, the second indication information, the third indication information, and the fourth indication information, respectively, to obtain a first exponential function value, a second exponential function value, a third exponential function value, and a fourth exponential function value; Determining weights respectively for the first exponential function value, the second exponential function value, the third exponential function value, and the fourth exponential function value in real time; Using respective weights, weighting the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value to obtain first aid indication information; Determine the vital sign corresponding to the first aid indication information, and judge whether the vital sign is a preset first aid sign.

6. The method according to claim 5, characterized in that The exponential function value is calculated using an exponential function, and the exponential function is expressed as: X 10*X ， Wherein, X is the first indication information, the second indication information, the third indication information or the fourth indication information.

7. The method according to claim 5, characterized in that The step of determining weights of the first exponential function value, the second exponential function value, the third exponential function value, and the fourth exponential function value in real time respectively includes: Determine the subcutaneous blood volume of the image at the last playback time in the facial video; According to the subcutaneous blood volume, weights of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value are determined.

8. The method according to claim 7, characterized in that Different subcutaneous blood volume ranges correspond to different weights of the first exponential function value, the second exponential function value, the third exponential function value and the fourth exponential function value; Determining the weights of the first exponential function value, the second exponential function value, the third exponential function value, and the fourth exponential function value according to the subcutaneous blood volume includes: A subcutaneous blood volume range within which the subcutaneous blood volume is located is determined, and weights are respectively determined for the first exponential function value, the second exponential function value, the third exponential function value, and the fourth exponential function value according to the subcutaneous blood volume range.

9. The method according to claim 5, characterized in that The determining of the vital signs corresponding to the first aid indication information and judging whether the vital signs are preset first aid signs include: Determine, according to the first aid indication information, the vital sign values ​​corresponding to the face, wherein the vital sign values ​​include a heart rate value and a blood pressure value; Determine whether the vital sign value is within a preset emergency sign range; If the vital sign value is within the preset range of the vital sign to be urgently needed, the vital sign is the preset vital sign to be urgently needed.

10. An online vital sign monitoring device based on rPPG emergency treatment, characterized in that: include: An acquisition unit, configured to acquire a facial video of a face to be detected; a determination unit configured to determine a change image of subcutaneous blood volume for the face presented in the facial video, wherein the change image includes an image before the change and an image after the change; a judgment unit configured to process the changed image using the rPPG to judge whether the vital sign corresponding to the face is a preset emergency sign; The generating unit is configured to generate and output a message for emergency treatment if the vital sign is a preset emergency treatment sign.

Images