Physiological condition detection method, device, electronic device, storage medium, and program

Abstract

The embodiments of the present disclosure provide a physiological state detection method, device, electronic device, storage medium, and program. The method includes the steps of acquiring a video stream collected by a photographing device, extracting a face image of a target object of multiple frames from the image of multiple frames in the video stream, determining at least one smooth region in the face image, performing weighting processing or region selection on the at least one smooth region based on region attribute information to obtain information waiting to be extracted regarding the physiological state, and performing physiological state information extraction on the information waiting to be extracted to obtain a physiological state detection result of the target object. In the embodiments of the present disclosure, physiological state detection is realized based on an image processing method, and can be measured in real time anytime and anywhere, which is practical. Moreover, the accuracy of the measurement can be improved by the constraint of a certain condition.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) The present disclosure claims priority to four Chinese patent applications filed on March 31, 2022 by Shanghai SenseTime Linggang Intelligent Technology Co., Ltd. with the China Patent Office, the entire contents of which are incorporated herein by reference. The four Chinese patent applications are: Chinese patent application having application number 202210344726.1 and the title of the invention "Physiological State Detection Method, Apparatus, Electronic Device, and Storage Medium", Chinese patent application having application number 202210344718.7 and the title of the invention "Physiological State Detection Method, Apparatus, Electronic Device, and Storage Medium", Chinese patent application having application number 202210346427.1 and the title of the invention "Blood Pressure Measurement Method, Apparatus, Electronic Device, and Storage Medium", and Chinese patent application having application number 202210346417.8 and the title of the invention "Physiological State Detection Method, Apparatus, Electronic Device, and Storage Medium".

[0002] The present disclosure relates to the field of computer technology, and to a physiological condition detection method, device, electronic device, storage medium, and program. [Background technology]

[0003] Accurate physiological state data is the basis for analyzing human body variability, so that the detection of physiological state is of great significance. Taking safe driving as an example, effective physiological state detection can help to grasp the physiological state of the passengers in the car, thereby providing auxiliary measures for safe driving. In the related art, physiological state detection is mainly performed by dedicated detection devices, such as blood pressure monitors, heart rate monitors, blood oxygen concentration monitors, etc. In addition, physiological state measurement can also be realized by wearable devices such as smart watches and smart handheld devices that integrate related sensor components.

[0004] Therefore, the above detection solutions require contact measurement by dedicated devices, which is inconvenient for detection, and therefore cannot fully meet the needs of scenes such as driving safety. Summary of the Invention

[0005] The embodiments of the present disclosure provide at least a physiological condition detection method, an apparatus, an electronic device, a storage medium, and a program.

[0006] An embodiment of the present disclosure provides a physiological state detection method, which includes the steps of: acquiring a video stream collected by a photographing device, extracting a face image of a target object of multiple frames from images of multiple frames in the video stream, determining at least one smooth region in the face image, performing weighting or region selection on the at least one smooth region according to region attribute information to obtain information to be extracted about a physiological state, and performing physiological state information extraction on the information to be extracted to obtain a physiological state detection result of the target object.

[0007] An embodiment of the present disclosure further provides a physiological state detection device, which includes: an acquisition unit configured to acquire a video stream collected by a photographing device; an extraction unit configured to extract a face image of a target object of multiple frames from images of multiple frames in the video stream; a determination unit configured to determine at least one smooth region in the face image; a processing unit configured to perform weighting or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding a physiological state; and a detection unit configured to perform physiological state information extraction on the information to be extracted to obtain a physiological state detection result of the target object.

[0008] An embodiment of the present disclosure further provides an electronic device including a processor, a memory, and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is in operation, the machine-readable instructions, when executed, causing the processor to perform the physiological condition detection method described above.

[0009] An embodiment of the present disclosure further provides a computer-readable storage medium having stored thereon a computer program which, when executed, causes a processor to perform the physiological condition detection method described above.

[0010] An embodiment of the present disclosure further provides a computer program comprising computer readable code, which when executed by an electronic device causes a processor of the electronic device to execute the physiological condition detection method described above. Effect of the Invention

[0011] When a video stream is obtained by applying the physiological state detection method, device, electronic device, storage medium, and program provided in the embodiments of the present disclosure, a face image of a target object in a plurality of frames can be extracted from the video stream, and at least one smooth region in the face image of each frame can be determined. Then, weighting or selection can be performed on the determined at least one smooth region based on the region attribute information of the smooth region, and processing can be performed based on the weighting or selection result to obtain the physiological state detection result of the target object. In the related art, it is necessary to perform contact measurement by a dedicated device, which is inconvenient for measurement. Compared with the problems in the related art, in the present disclosure, physiological state detection is realized based on an image processing method, and measurement can be performed in real time anytime and anywhere, which is practical. In this way, the accuracy and flexibility of measurement can be improved.

[0012] In order to make the above objects, features and advantages of the present disclosure more clear and understandable, preferred embodiments are described in detail below with reference to the drawings. [Brief description of the drawings]

[0013] [Figure 1] 2 is a flowchart of a physiological condition detection method according to a first aspect provided in an embodiment of the present disclosure. [Figure 2A] 1 is a schematic diagram of face feature points that can be extracted from a face image captured by a camera. [Figure 2B] 4 is a flowchart of a physiological condition detection method according to a second aspect provided in an embodiment of the present disclosure. [Figure 2C] 1 is a flowchart of a physiological condition detection method according to a third aspect provided in an embodiment of the present disclosure. [Figure 2D] 11 is a flowchart of a physiological condition detection method according to a fourth aspect provided in an embodiment of the present disclosure. [Figure 2E] 11 is a flowchart of a physiological condition detection method according to a fifth aspect provided in an embodiment of the present disclosure. [Diagram 3] 1 is a schematic diagram of a physiological condition detection device provided in an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to more clearly explain the technical solutions of the embodiments of the present disclosure, the drawings necessary for use in the embodiments of the present disclosure are described above. The drawings herein are incorporated in and constitute a part of the specification. These drawings show embodiments consistent with the present disclosure and are intended to explain the technical solutions of the present disclosure together with the specification.

[0015] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will clearly and completely describe the technical solutions of the embodiments of the present disclosure with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all of the embodiments. Usually, the components of the embodiments of the present disclosure described and disclosed in the drawings herein may be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present disclosure provided in the drawings below is not intended to limit the protection scope of the present disclosure as claimed, but merely represents selected embodiments of the present disclosure. All other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without any creative effort shall be included in the protection scope of the present disclosure.

[0016] It should be noted that like labels and letters represent like items in the following drawings, so that if an item is defined in one drawing, it need not be further defined or interpreted in any subsequent drawings.

[0017] The term "and / or" below only represents a relational relationship and indicates that three types of relationships can exist. For example, "A and / or B" can represent that only A exists, that A and B exist simultaneously, or that only B exists. In addition, the term "at least one type" below represents any one type among a plurality of types, or any combination of at least two types among a plurality of types. For example, "including at least one type of A, B, and C" can represent that any one or more elements selected from the set consisting of A, B, and C are included.

[0018] As can be seen from research, in the related art, physiological state detection is mainly performed by dedicated detection devices, such as blood pressure monitors, heart rate monitors, blood oxygen concentration monitors, etc. In addition, physiological state measurement can also be realized by wearable devices such as smart watches and smart handhelds that integrate related sensor components. Therefore, the above detection solutions require contact measurement by dedicated devices, which is inconvenient for detection, and therefore cannot fully meet the needs of situations such as safe driving.

[0019] To solve the above problems, the related art provides a non-contact physiological condition detection solution, namely, remote photoplethysmography (rPPG). This method can perform detection using currently widely used mobile phone terminals with cameras, and does not require additional hardware costs, making it easy to use. However, the current bottleneck of the rPPG method is that the detection accuracy is lower than that of some dedicated detection devices, and it is easily affected by external light. In addition, in the detection of physiological characteristics using the rPPG method, the object to be detected needs to be stationary for a certain period of time, and it can only be used for spontaneous detection.

[0020] In conventional rPPG-based physiological feature detection, it is necessary to select a Region of Interest (ROI). In actual camera imaging, the screen may be too dark or too bright. For a single ROI region, the range of one face is often fixed as the ROI region, and there is a tendency to select an area with less than ideal exposure for signal extraction. Or, in actual camera imaging, there is an angle between the face and the camera optical axis, so the pixel area occupied by the ROI region after imaging may be too small. Therefore, the amount of information in the ROI region is small, which makes it difficult to obtain ideal detection results.

[0021] Based on the above research, the embodiment of the present disclosure provides a solution to realize physiological state detection under the constraint of a certain condition, for example, by selecting an optimal ROI region based on image brightness, or by weighting based on region area, or by weighting based on region area and region brightness to realize physiological state detection, thereby obtaining a signal amount closer to the actual situation and improving detection accuracy.

[0022] In order to facilitate understanding of the present embodiment, the physiological state detection method disclosed in the embodiment of the present disclosure will be introduced in detail first. The execution entity of the physiological state detection method provided in the embodiment of the present disclosure is usually an electronic device having a certain computing capability. The electronic device includes, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, an in-vehicle device, a wearable device, etc. In some embodiments of the present disclosure, the physiological state detection method may be realized by a processor calling computer-readable instructions stored in a memory.

[0023] Referring to Figure 1, Figure 1 is a flowchart of a physiological state detection method provided in an embodiment of the present disclosure. The method includes steps S101 to S105.

[0024] In step S101, a video stream collected by a photographing device is obtained.

[0025] In step S102, multi-frame face images of a target object are extracted from the multi-frame images in the video stream.

[0026] In step S103, at least one smooth region in the surface image is determined.

[0027] In step S104, weighting or region selection is performed on at least one smooth region based on the region attribute information to obtain information awaiting extraction regarding the physiological state.

[0028] In step S105, physiological state information extraction is performed on the information waiting to be extracted, and a physiological state detection result of the target subject is obtained.

[0029] The physiological state detection method in the embodiment of the present disclosure can be applied to the automotive field where physiological state detection is required. The embodiment of the present disclosure can realize physiological state detection for a human body in a vehicle cabin environment. Furthermore, when a physiological state detection result for a target object in a vehicle cabin environment is obtained based on the physiological state detection method provided in the embodiment of the present disclosure, it can quickly grasp whether the target object is in an abnormal physical state, and if the physical state is abnormal, it can quickly issue a warning or provide assistance. In this way, it is possible to provide a sufficient possibility of safe driving.

[0030] In addition, the embodiments of the present disclosure can also be applied to any other related fields that need to perform physiological state detection, such as medical care, home life, etc. Considering that the automobile driving field is widely applied, the following description often takes the automobile field as an example. Furthermore, the video stream in the embodiments of the present disclosure can be collected and obtained by a shooting device (e.g., a fixed camera inside the car in the automobile scene), or collected and obtained by a camera attached to a user terminal, or collected and obtained in other ways.

[0031] To realize physiological state detection for a specific target object, the installation position of the camera can be preset based on the specific target object. For example, to realize physiological state detection for a driver in a vehicle, the camera can be installed in a position where the shooting range covers the driving area, such as inside the A-pillar of the vehicle, on the console, or at the steering wheel. Or, to realize physiological state detection for passengers with multiple riding attributes including the driver and passengers in a vehicle, the camera can be installed in a position where the shooting range covers multiple seat areas in the vehicle, such as the rearview mirror, top accessories, reading lights, etc.

[0032] In practical applications in the automotive field, an in-vehicle image collection device included in a Driver Monitoring System (DMS) can be further adopted to realize the collection of video streams of relevant driving areas, or an in-vehicle image collection device included in an Occupant Monitoring System (OMS) can be adopted to realize the collection of video streams of relevant riding areas.

[0033] Considering that changes in skin color and brightness due to blood vessel flow in the face can reflect physiological states such as heart rate and breathing, face detection can be first performed on multiple frame images in the video stream, multiple frame face images of the target object in the vehicle cabin can be extracted, and then physiological state information can be extracted from the face images.

[0034] The target object is a measurement object for detecting a physiological state of the human body, and may be a person object that stops at a predetermined position, or a person object that has a specific identity attribute. For example, in the automobile field, the target object may be an object with a specific riding attribute, such as a driver or a passenger in the front seat. Or, the target object may be an object whose identity mark is registered in advance using face information, such as an owner registered by an application program. Or, the target object may further be any passenger in the vehicle. At least one passenger may be located by performing face detection on the video stream in the vehicle cabin, and the detected one or more passengers may be the target object.

[0035] In the process of performing face detection, multiple target faces may appear in one frame image. In some situations, it is possible to select to perform physiological state detection for an occupant at a certain riding position, i.e., the occupant at the riding position is taken as a target object. In order to realize physiological state detection for a target object in a vehicle cabin, multiple frame face images of the target object can be determined from the detected face images based on the face detection results of the multiple frame images and a predetermined riding position, where the predetermined riding position is for indicating the position of the target object to be measured.

[0036] The camera that collects the video stream in the vehicle cabin has a fixed relative position in the vehicle interior space, and the collected image can be divided into seat areas based on the position of the camera. For example, for a five-seat private car, an image area can be divided into an image area corresponding to the driver's seat, an image area corresponding to the passenger seat, an image area corresponding to the left rear seat, an image area corresponding to the right rear seat, and an image area corresponding to the middle rear seat. Based on the position in the image of the face of each passenger object in the vehicle and the coordinate range of each image area, an image area that contains the face of each passenger object can be determined, and thereby the passenger object at a predetermined riding position is determined as a target object.

[0037] In practical application, the OMS usually takes images in the entire vehicle interior space, and may take images of multiple people. The "front seat" and "rear seat" can be manually selected to specify the target object to be measured in the vehicle interior space, and in this case, the embodiment of the present disclosure can perform measurement on the face of the corresponding area in the image. If the DMS takes images in the main driving area and the captured object only includes the driver, there is no need to specify the object.

[0038] In some embodiments of the present disclosure, physiological conditions, such as heart rate, respiratory rate, blood oxygen, blood pressure, etc., usually need to be monitored for a certain length of time for evaluation, so in the embodiments of the present disclosure, image change information corresponding to multiple frames of face images in a video stream that lasts for a certain period of time is adopted to realize extraction of physiological condition information, so that the physiological condition detection result thus extracted is more in line with the needs of actual scenes.

[0039] In some embodiments of the present disclosure, in the process of analyzing image change information based on a face image, there is a problem that detection cannot be continued due to the influence of face rotation, occlusion, or loss of ROI area (i.e., corresponding to the smooth area in the physiological state detection method provided in the embodiments of the present disclosure) caused by external light, so that the embodiments of the present disclosure can use multiple smooth areas in a face image to perform detection. In this way, even if one of the smooth areas is lost, detection can be continued if other smooth areas are better, thereby making the application scenarios wider.

[0040] In some embodiments, considering the existence of an angle between the face and the camera optical axis, different smooth regions occupy different pixel areas in the image after the camera is focused. Therefore, the embodiments of the present disclosure can further perform weighting according to the selected smooth regions and the pixel areas of the smooth regions in the screen, so as to obtain a signal amount closer to the actual situation, thereby improving the detection accuracy.

[0041] In some embodiments, furthermore, considering the existence of an angle between the face and the camera optical axis, different smooth regions occupy different pixel areas and corresponding pixel brightness in the image after the camera is focused, so that weighting can be performed according to the pixel areas and corresponding pixel brightness in the screen of the smooth regions, so as to obtain a signal amount closer to the actual situation, thereby improving the detection accuracy.

[0042] Here, at least one smooth region can be determined for the face image of each frame, and the smooth region can be determined by performing selection on one or more face smooth sub-regions in the face image. Here, the face smooth sub-region can be a smooth continuous region, and the continuous region can be required to have a predetermined shape, such as a rectangle, a circle, or an ellipse, that can be located by face keypoints. Each continuous region does not include non-smooth features, such as the eyes, nose, mouth, eyebrows, etc. of the face. The continuous region has a more average reflectance to some extent, thereby more effectively capturing the skin color and brightness changes caused by the blood flow of the face, and then realizing more accurate physiological state detection.

[0043] When multiple surface smooth sub-regions of the surface image of each frame are determined, the physiological state detection method provided in the embodiment of the present disclosure can first determine at least one smooth region based on a predetermined condition set for image brightness, and then extract physiological state information based on image change information corresponding to the image information of the at least one smooth region. The extracted physiological state detection result can be a detection result including at least one of heart rate, breathing frequency, blood oxygen, blood pressure, etc.

[0044] In an embodiment of the present disclosure, a ROI region that is optimal for the image brightness of multiple surface smooth sub-regions is selected as a smooth region, and photoplethysmography (PPG) signal extraction is performed on the smooth region, thereby improving the detection accuracy and detection interference resistance of the rPPG method.

[0045] Here, the predetermined condition set for image brightness can be a condition for removing surface smooth sub-regions related to obvious overexposure or obvious too dark. Considering that the image brightness corresponding to the obviously overexposed surface smooth sub-region is abnormally high, the accuracy of the PPG signal extraction performed in this case is not sufficient, so the detection accuracy of the finally obtained physiological state detection result is low. Similarly, for the obviously too dark surface smooth sub-region, its image brightness is abnormally low, which also leads to low detection accuracy.

[0046] In some embodiments of the present disclosure, first, a face image can be subjected to face feature extraction, and then at least one smooth region in the face image can be determined based on the extracted face feature points. That is, the above step S103 can be realized by the following process.

[0047] In step 1, face feature extraction is performed on a face image to obtain the positions of a number of pre-set key feature points in the face image.

[0048] In step 2, based on the locations of the plurality of preset key feature points, at least one smooth region in the face image corresponding to the plurality of preset key feature points is determined.

[0049] Here, the process of extracting the face feature points can be realized by using a face keypoint detection algorithm. For example, face feature points for a standard face image can be preset, where the standard face can be a face image including the five senses and photographed facing forward by a camera. In this way, in the process of extracting face feature points for the face image of the target object extracted from the image of each frame, each face feature point can be determined based on a comparison situation between the extracted face image of the target object and the standard face image. For example, the face feature points can be related features having obvious face characteristics, such as eyebrow feature points, nose bridge feature points, nose tip feature points, cheek feature points, and mouth feature points.

[0050] In an embodiment of the present disclosure, one or more smooth regions in a face image can be determined based on the positions of a plurality of preset key feature points in the face image. Here, the smooth region can be a rectangular region or other region having a continuous shape, and the embodiment of the present disclosure is not limited thereto. In the following, a roughly rectangular region is taken as an example.

[0051] In practical application, the smooth regions here may be a forehead smooth region determined based on eyebrow feature points, a left upper cheek smooth region and a right upper cheek smooth region determined based on cheek feature points, nose bridge feature points and nose tip feature points, and a left lower cheek smooth region and a right lower cheek smooth region determined based on cheek feature points, nose tip feature points and mouth feature points.

[0052] In some embodiments of the present disclosure, if the regions are not occluded, the above five regions (forehead smooth region, left upper cheek smooth region, right upper cheek smooth region, left lower cheek smooth region and right lower cheek smooth region) can be extracted simultaneously from one frame of face image.

[0053] At the same time, when an area is occluded, the area that can actually be extracted from a frame of face image can be determined according to the actual situation. In the embodiment of the present disclosure, the occluded smooth area can be determined according to the following steps:

[0054] In step 1, if a pre-defined key feature point is missing among the extracted face feature points, the face sub-region corresponding to the missing pre-defined key feature point is determined as a missing region, and other face sub-regions among the pre-defined multiple face sub-regions other than the missing region are determined as non-missing regions.

[0055] In step 2, a boundary of the non-missing region is determined based on the positions of preset key feature points corresponding to the non-missing region among the extracted surface feature points, thereby determining at least one smooth region.

[0056] In some embodiments of the present disclosure, the absence of a preset key feature point in an extracted face feature point indicates that the corresponding face sub-region is occluded, and the face sub-region corresponding to other non-missing face key points can be determined to be a non-missing region. If the boundary of the non-missing region is determined based on the position of the preset key feature point corresponding to the non-missing region, it can be said that a smooth region is determined. It can be seen that even if the region is occluded, the smooth region can be extracted relatively accurately based on the above method in the embodiments of the present disclosure.

[0057] As shown in FIG. 2A, FIG. 2A is a schematic diagram of face feature points that can be extracted from a face image captured by a camera. Here, a total of 106 feature points are extracted from the face. Here, five smooth regions can be selected based on the coordinate information of the face feature points. Referring to FIG. 2A, here, region 1 can be a rectangular ROI of region 1 configured by two feature points of eyebrows on both sides. Region 2 is one cheek region on the left side, and the rectangular ROI of region 2 can be configured by the positions of the edge feature point, nose ridge feature point, and left eye feature point on the left side of the face. Region 3 is one cheek region on the right side, and the rectangular ROI of region 3 can be configured by the positions of the edge feature point, nose ridge feature point, and right eye feature point on the right side of the face. Region 4 is another cheek region on the left side, and the rectangular ROI of region 4 can be configured by the positions of the edge feature point, left nostril feature point, and left mouth feature point on the left side of the face. Region 5 is another cheek region on the right side, and a rectangular ROI for region 5 can be configured by the positions of the right face edge feature point, the right nostril feature point, and the right mouth feature point.

[0058] The embodiment of the present disclosure can also be realized by combining face pose in the process of performing face feature extraction, in which the face pose of the target object can be first obtained based on face image detection, and then face feature extraction can be performed on the face image based on the face pose.

[0059] Here, the face posture can be determined based on a pre-trained face posture detection network. The face posture detection network can be a neural network of any structure, for example, a multi-layer convolution neural network, and can learn the correspondence between the face image sample and the face posture corresponding to the label. Here, the labelable face posture includes information such as the rotation direction and rotation angle of the head relative to the camera. In this way, the face posture of the face can be determined by inputting the face image to the trained face posture detection network. For example, the face may be tilted 45° to the left. In this case, the image of the face tilted 45° to the left can be compared with a standard face image tilted to the left to extract the related pre-set key feature points, which is simple to operate.

[0060] In the embodiment of the present disclosure, the areas of different face sub-regions shown in the image by different face postures may be different, and the areas of the smooth regions are also different correspondingly. Still, take the face tilted 45° to the left as an example, in this case, the left cheek region has a small area in the image or is barely visible, and the area of ​​the corresponding smooth region is close to 0.

[0061] The above-mentioned region attribute information may be information on at least one attribute of the smooth region, such as brightness, area, location, etc. In the embodiment of the present disclosure, the weight of each smooth region can be calculated or a selection process can be performed based on the above-mentioned region attributes of each smooth region, and the image information of the smooth region related to the physiological state can be obtained as the information to be extracted after the weighting or selection process. Alternatively, after the weighting is calculated, a time domain signal extraction operation is performed, and the signal obtained by weighting the time domain signal corresponding to each smooth region is the information to be extracted related to the physiological state.

[0062] In some embodiments of the present disclosure, when the region attribute information includes region brightness, the information to be extracted regarding the physiological state includes image information of at least one region of interest. In this case, it is possible to realize performing region selection on at least one smooth region based on the region brightness to obtain at least one region of interest. Furthermore, it is possible to perform physiological state information extraction on the image information of the at least one region of interest to obtain a physiological state detection result. As shown in FIG. 2B, FIG. 2B is a flowchart of a physiological state detection method of a second aspect provided in an embodiment of the present disclosure. Step S104 provided in an embodiment of the present disclosure can be realized by the following step S201.

[0063] In step S201, at least one region of interest whose region luminance satisfies a predetermined condition is determined from at least one smooth region.

[0064] Here, step S105 provided in the above embodiment can be realized by the following step S202.

[0065] In step S202, physiological state information extraction is performed on image information of at least one region of interest to obtain a physiological state detection result.

[0066] In some embodiments of the present disclosure, realizing the selection of relevant regions of interest based on an image brightness threshold can be realized by the following steps.

[0067] In step 1, an image brightness threshold corresponding to the surface image and the region image brightness of each smooth region are determined.

[0068] In step 2, filtering is performed on at least one smooth region based on an image brightness threshold and a region image brightness to obtain at least one candidate smooth region.

[0069] In step 3, at least one region of interest whose region intensity satisfies a preset intensity condition is selected from the at least one candidate smooth region.

[0070] Here, first, the smooth regions are filtered based on an image brightness threshold, and then a region of interest with the optimal brightness can be selected based on the candidate smooth regions obtained by filtering.

[0071] To achieve effective filtering, in the embodiment of the present disclosure, the limitation can be performed by an image brightness threshold of maximum brightness, and the limitation can also be performed by an image brightness threshold of minimum brightness. The former can remove smooth regions whose regional image brightness exceeds a first preset proportion of the maximum brightness, for example, the smooth regions whose regional image brightness exceeds 90% of the maximum brightness, thereby removing relevant smooth regions that may be overexposed. The latter can remove smooth regions whose regional image brightness does not reach a second preset proportion of the minimum brightness, for example, the smooth regions whose regional image brightness is less than 30% of the minimum brightness, thereby removing relevant smooth regions that may be too dark.

[0072] Here, the region image luminance described above is a numerical value, which may represent the average value of the region luminance, i.e., a numerical value obtained by adding up and averaging the pixel luminance values ​​of all pixels in the region, or may represent the maximum pixel luminance value of all pixels in the region, or may represent the minimum pixel luminance value of all pixels in the region, or may be a numerical value obtained by performing a calculation on the pixel luminance values ​​of all pixels in the region according to a preset calculation method.

[0073] In some embodiments of the present disclosure, the maximum brightness and the minimum brightness may be preset brightness values, and different brightness values ​​may be set for different application scenarios and / or different image formats. In practical applications, they may be adjusted based on needs. Here, considering that within a certain brightness range, the higher the brightness, the more likely the area has distinct image features, and thus the physiological detection result obtained is more accurate. Based on this, in the embodiments of the present disclosure, the candidate smooth area with the maximum brightness may be selected as the smooth area.

[0074] In the embodiment of the present disclosure, when there are multiple smooth regions determined, the luminance weight of each smooth region can be determined first, and then fusion can be performed on the physiological state information extraction results of the multiple smooth regions based on the luminance weight. Here, the associated luminance weight can be determined based on the region image luminance of the corresponding region. For example, a smooth region with a larger region image luminance can correspond to a larger luminance weight, and conversely, a smooth region with a smaller region image luminance can correspond to a smaller luminance weight. Thereby, the influence of the smooth region with a larger luminance on the physiological state detection can be increased, and the influence of the smooth region with a smaller luminance on the physiological state detection can be weakened. In this way, the physiological state detection result obtained by the fusion can be more accurate.

[0075] In some embodiments of the present disclosure, different smooth regions may be affected by different external factors (e.g., illumination, etc.), and thus the image features affected by the physiological state that each of them exhibits are different. In view of this, in order to capture as many effective surface features as possible and realize more accurate physiological state detection, when at least one region of interest whose region luminance satisfies a predetermined condition is determined based on at least one candidate smooth region, physiological state information extraction can be performed based on image information of the region of interest, where the image information of the region of interest includes pixel luminance information of three different color channels. Then, physiological state information extraction is performed on the image information of the at least one region of interest to obtain a physiological state detection result of the target object. That is, the above step S202 can be realized by the following steps.

[0076] In step 1, a time-domain signal representative of the physiological state of a target object is determined based on pixel intensity information of three color channels of a region of interest.

[0077] In step 2, a frequency domain transformation is performed on the time domain signal to obtain a frequency domain signal representing the physiological state of the target subject.

[0078] In step 3, a physiological state value of the target object is determined based on the peak value of the frequency domain signal.

[0079] In some embodiments of the present disclosure, considering that the physiological state directly affects the blood flow change of the target object, and the blood flow change affects the brightness change of the image, here, first, a time domain luminance signal corresponding to each color channel among the three color channels of red, green, and blue of the region of interest can be determined to form an RGB three-dimensional signal, and then a principal component analysis is performed on the time domain luminance signals of the three different color channels, and the one-dimensional signal obtained after extracting the principal components (dimension reduction) is the time domain signal representing the physiological state of the target object. The time domain signal can be determined by the time domain luminance signal of one channel (e.g., the green channel) among the above color channels, and the selected channel can be the one channel that can best represent the blood flow change. That is, the step of "determining a time domain signal representing the physiological state of the target object based on pixel luminance information of the three color channels of the region of interest" in the above step 1 can be realized by the following process.

[0080] First, based on the pixel luminance information of the three color channels corresponding to the region of interest, the time-domain luminance signals corresponding to each color channel of the smooth region are determined.

[0081] Then, a principal component analysis is performed on the time-domain luminance signals corresponding to the three different color channels of the region of interest to obtain the time-domain signals.

[0082] In some embodiments, in the process of determining the time domain luminance signal corresponding to each color channel of each region of interest, image change information corresponding to multiple frames of face images in a video stream of a certain duration is required, because physiological status information such as heart rate, breathing frequency, blood oxygen, blood pressure, etc. require detection of a certain time length, and in the process of determining the time domain luminance signal corresponding to the region of interest, luminance values ​​of multiple color channels can be utilized, thereby improving the accuracy of the time domain luminance signal extraction.

[0083] Here, in order to realize a more accurate principal component analysis, processes such as normalization and de-trend filter noise removal can be performed on the three-dimensional time domain luminance signal before performing the principal component analysis. In addition, after the principal component analysis, the obtained time domain signal can be further subjected to a noise removal process of moving average filtering. This can improve the accuracy of the time domain signal and improve the accuracy of the physiological state detection result obtained in the subsequent processing.

[0084] In some embodiments of the present disclosure, considering the existence of an angle between the face and the camera optical axis, different smooth regions occupy different pixel areas in the image after the camera is focused, so that weighting can be performed according to the selected at least one smooth region and the pixel area of ​​the smooth region in the screen, so as to obtain a signal amount closer to the actual situation and improve the detection accuracy.

[0085] When the smooth regions of the face image are determined, the physiological state detection method provided in the embodiment of the present disclosure can first determine a time domain luminance signal corresponding to each smooth region, and then perform weighting processing on each time domain luminance signal based on the area of ​​each smooth region, thereby conveniently performing physiological state information extraction based on the area-weighted time domain luminance signal. Here, the extracted physiological state detection result may be a detection result including at least one of heart rate, breathing frequency, blood oxygen, blood pressure, etc. Next, when the time domain luminance signal corresponding to each smooth region is determined, weighting processing can be performed on the time domain luminance signal determined based on the area of ​​the smooth region. The area-weighted time domain luminance signal can increase the influence of larger smooth regions in the face image on physiological state detection and weaken the influence of smaller smooth regions in the face image on physiological state detection, thereby improving the accuracy of detection. In addition, the entire detection process does not require specialized equipment, and can be measured anytime and anywhere, which is practical.

[0086] In some embodiments of the present disclosure, when the region attribute information includes a region area, the information to be extracted regarding the physiological state may include a weighted time domain luminance signal. In this case, a weighting process may be performed on the time domain luminance signal corresponding to at least one smooth region based on the region area to obtain an area-weighted time domain luminance signal. Then, physiological state information extraction may be performed based on the area-weighted time domain luminance signal to obtain a physiological state detection result. As shown in FIG. 2C, FIG. 2C is a flowchart of a physiological state detection method according to a third aspect provided in an embodiment of the present disclosure. Step S104 provided in an embodiment of the present disclosure may be realized by the following steps 203 and S204.

[0087] In step S203, for each smooth region, a time-domain luminance signal corresponding to the smooth region is generated based on pixel luminance information of at least one color channel of the smooth region in multiple frames of face images.

[0088] In step S204, a weighting process is performed on the time domain luminance signal corresponding to at least one smooth region based on the area of ​​each smooth region to obtain an area-weighted time domain luminance signal.

[0089] Here, step S105 provided in the above embodiment can be realized by the following step S205.

[0090] In step S205, physiological state information is extracted based on the area-weighted time domain luminance signal to obtain a physiological state detection result.

[0091] In some embodiments of the present disclosure, when each smooth region with a relatively strong signal expression ability is determined, then, for each smooth region, a time domain luminance signal corresponding to each color channel of the smooth region can be determined according to the luminance values ​​corresponding to the three color channels of the smooth region in the surface images of multiple frames. In this way, for each color channel, a weighting process is performed on the time domain luminance signal in the color channel of at least one smooth region according to the area of ​​each smooth region, to obtain a time domain luminance signal after area weighting in the color channel. That is, the embodiment of the present disclosure realizes the area weighting in each color channel, so that each color channel has a stronger expression ability of blood flow change. That is, step S203 provided in the embodiment of the present disclosure can be realized by the following process.

[0092] Based on pixel luminance information corresponding to three color channels of a smooth region in multiple frames of face images, a time domain luminance signal corresponding to each color channel of the smooth region is determined.

[0093] Correspondingly, step S204 provided in the above embodiment can be realized by the following process.

[0094] For each color channel, a weighting process is performed on the time-domain luminance signal in the color channel of at least one smooth region based on the area of ​​each smooth region to obtain an area-weighted time-domain luminance signal in the color channel.

[0095] At the same time, step S205 provided in the above embodiment can be realized by the following process.

[0096] In step 1, a principal component analysis is performed on the area-weighted time-domain luminance signals in multiple different color channels of the smooth region to obtain a time-domain signal representing the physiological state of the target object.

[0097] In step 2, frequency domain processing is performed on the time domain signal representing the physiological state of the target object, and the physiological state detection result is determined based on the peak value of the frequency domain signal obtained by the frequency domain processing.

[0098] In some embodiments, considering that different area sizes mean different amounts of physiological feature information contained in the corresponding smooth areas, here, the area weight of the corresponding smooth areas can be determined based on the area of ​​each smooth area, and weighting processing can be performed on the time domain luminance signal corresponding to at least one smooth area according to the area weight, thereby obtaining a signal amount closer to the actual situation and improving the accuracy of the generated physiological state detection result.

[0099] In the embodiment of the present disclosure, different face postures show different face sub-regions in the image with different areas, and the areas of the corresponding smooth regions are also different. Still, take the face tilted 45° to the left as an example, in this case, the left cheek region is barely visible in the image, and the area of ​​the corresponding smooth region is 0.

[0100] In some embodiments of the present disclosure, the area of ​​each smooth region can be further estimated based on the face posture and the parameters of the photographing device. For example, after completing the detection of the face posture by a neural network or the like, a face model facing the camera is projected into an image coordinate system based on parameters such as the face posture angle and the camera focal length, and the coordinates of the boundary points in the image coordinate system of each smooth region located based on the face key points are calculated, thereby estimating the area value of each smooth region in the image. When some areas of the face are insufficiently bright or some key points are missing, the area value can be the area for estimating the area of ​​each smooth region in the image. In the embodiments of the present disclosure, the area of ​​the smooth region can be determined based on the position of a preset key feature point corresponding to each smooth region. In practical application, the position of each preset key feature point can be the vertex position of the smooth region, and the length and width of the smooth region can be determined according to the vertex distance, thereby determining the area of ​​the region. The area of ​​the region determined in this way is more accurate.

[0101] A smooth region with a larger area can contain more information to some extent, and can be assigned a larger area weight. Conversely, a smooth region with a smaller area can contain less information to some extent, and can be assigned a smaller area weight. In an embodiment of the present disclosure, the area weight corresponding to each smooth region can be determined and area weighted based on the proportion of each region area to the total region area.

[0102] Before performing area weighting, in order to ensure the extraction of as much information as possible, when at least one smooth region is determined, the smooth region whose area is smaller than a threshold is removed. Here, the threshold can be a preset fixed value, or a value determined based on the area distribution situation of each smooth region in the entire face region in the current image. For example, the threshold can be a predetermined percentage (e.g., 20%) of the maximum area of ​​each smooth region in the current image.

[0103] In practical application, the smooth regions whose visible range in the image does not meet the preset visibility requirement can be removed from the smooth regions based on the face posture. Still, take the face tilted 45° to the left as an example, in this case, the area of ​​the right cheek region is smaller than the preset threshold and is almost invisible in the image, in this case, the corresponding smooth region can be directly removed.

[0104] In the physiological state detection method provided in the embodiment of the present disclosure, when at least one smooth region of the face image is determined, if the region attribute information includes region area and region brightness, the extraction waiting information regarding the physiological state includes fusion image information. The fusion image information here represents image information fusing multiple dimensional image elements. In this case, a first contribution degree of each smooth region to the physiological state detection result and a second contribution degree of each smooth region to the physiological state detection result can be determined first, the first contribution degree represents the contribution degree of image information related to the region area in each smooth region to the physiological state detection result, and the second contribution degree represents the contribution degree of image information related to the region brightness in each smooth region to the physiological state detection result. Next, fusion is performed on the image information of the at least one smooth region based on the first contribution degree and the second contribution degree to obtain fusion image information. Finally, physiological state information extraction is performed based on the fusion image information, where the extracted physiological state detection result may be a detection result including at least one of heart rate, breathing frequency, blood oxygen, blood pressure, etc. In this way, when the first and second contributions of each smooth region to the physiological state detection result can be determined based on the area and pixel brightness information of each smooth region, respectively, the image information of at least one smooth region is fused, and finally, physiological state information is extracted based on the image information obtained by the fusion to obtain the physiological state detection result. In this way, in the process of detecting the physiological state, the area area and the brightness of each smooth region are combined to realize the fusion of the image information of each smooth region, thereby improving the accuracy of the measurement. As shown in FIG. 2D, FIG. 2D is a flowchart of a physiological state detection method according to a fourth aspect provided in the embodiment of the present disclosure. That is, step S104 provided in the above embodiment can be further realized by the following steps S206 to S208.

[0105] In step S206, a first contribution of each smooth region to the physiological condition detection result is determined based on the area of ​​each smooth region.

[0106] In step S207, a second contribution of each smooth region to the physiological state detection result is determined based on the pixel luminance information of each smooth region.

[0107] In step S208, fusion is performed on the image information of at least one smooth region according to the first contribution and the second contribution to obtain fused image information.

[0108] Here, step S105 provided in the above embodiment can be realized by the following step S209.

[0109] In step S209, physiological state information extraction is performed on the fusion image information to obtain a physiological state detection result.

[0110] Here, the relative fusion process may be a weighted fusion based on the first contribution and the second contribution, that is, for a smooth region, the higher the corresponding first contribution and second contribution, the stronger the representation ability of the image information after fusion corresponding to the smooth region.

[0111] Alternatively, the above fusion process can be further performed based on a comparison result between the first contribution and the second contribution. In some embodiments, when the first contribution of a smooth region is much larger than the second contribution, the physiological state information extraction result corresponding to the smooth region can be determined based on the luminance information of the smooth region, and the influence of the area can be ignored. When the first contribution of a smooth region is much smaller than the second contribution, the physiological state information extraction result corresponding to the smooth region can be determined based only on the area of ​​the smooth region, and the influence of the luminance can be ignored.

[0112] Here, the first contribution can be determined based on the area of ​​the corresponding smooth region, and the value of the first contribution is directly proportional to the area of ​​the corresponding smooth region. That is, a smooth region with a large area provides a high first contribution to the physiological state detection to a certain extent, and conversely, a smooth region with a small area provides a low first contribution to the physiological state detection. The reason is that as the area of ​​a region increases, the amount of effective information that can be extracted increases, and the increase in the amount of effective information can improve the accuracy of the physiological state detection to a certain extent.

[0113] Also, the associated second contribution can be determined based on pixel luminance information of the corresponding smooth region, and the second contribution is directly proportional to the average pixel luminance value of the corresponding smooth region. That is, a smooth region with strong pixel luminance provides a high second contribution to physiological state detection to a certain extent, and conversely, a smooth region with weak pixel luminance provides a low second contribution to physiological state detection. The reason is that as pixel luminance improves, the corresponding image quality improves, and better image quality can improve the accuracy of physiological state detection to a certain extent. Besides, the second contribution is inversely proportional to the variance of pixel luminance values ​​of the corresponding smooth region, and in this case, the corresponding second contribution can be determined based on the variance value.

[0114] In some embodiments, when determining the second contribution based on pixel luminance information, it is necessary to consider possible extreme situations of overexposure or too dark. Such extreme situations may affect the accuracy of physiological state detection. Based on this, in practical applications, the smooth area that is overexposed or too dark can be removed based on pixel luminance first, and then the second contribution can be determined for the smooth area that meets the luminance requirement, thereby ensuring the accuracy of the final physiological state detection result.

[0115] In some embodiments, when smooth regions are extracted from the face image of each frame, on the one hand, a first contribution corresponding to each smooth region can be determined based on the area of ​​each smooth region, and on the other hand, a second contribution corresponding to each smooth region can be determined based on the pixel brightness information of each smooth region.

[0116] Here, if the area of ​​the smooth region is smaller than the preset area threshold, the first contribution of the smooth region is determined to be 0, i.e., for smaller smooth regions with limited image display capability, the corresponding first contribution may not be considered here. Also, when determining the average luminance value of each smooth region based on the pixel luminance information of each smooth region, if the average luminance value is smaller than the first preset luminance threshold, the second contribution of the smooth region is determined to be 0, or if the average luminance value is larger than the second preset luminance threshold, the second contribution of the smooth region is determined to be 0. That is, for smooth regions that may be overexposed or too dark, the corresponding second contribution may not be considered here.

[0117] Based on this, if the area of ​​a smooth region is normal area and the smooth region is in normal brightness, the corresponding area weight and brightness weight can be determined respectively, where the area weight can be realized by the following steps:

[0118] In step 1, an addition operation is performed on the area of ​​each smooth region to obtain a total area value.

[0119] In step 2, for each smooth region, the ratio of the area of ​​the smooth region to the total area value is calculated to obtain the area proportion of the smooth region, which is the area weight corresponding to the smooth region.

[0120] Here, a sum can be obtained after the area of ​​each smooth region is summed. A larger area weight can be determined for a smooth region whose area is a large proportion of the total area value. Conversely, a smaller area weight can be determined for a smooth region whose area is a small proportion of the total area value.

[0121] Also, the associated luminance weights can be realized according to the following steps.

[0122] In step 1, an addition operation is performed on the average luminance value of each smooth region to obtain a luminance sum value.

[0123] In step 2, for each smooth region, the ratio between the average luminance value and the total luminance value of the smooth region is calculated to obtain the luminance proportion of the smooth region, which is the luminance weight corresponding to the smooth region.

[0124] Here, a luminance sum can be obtained after summing the average luminance values ​​of each smooth region. A larger luminance weight can be determined for a smooth region whose average luminance value of the smooth region accounts for a larger proportion of the luminance sum. Conversely, a smaller luminance weight can be determined for a smooth region whose luminance of the smooth region accounts for a smaller proportion of the luminance sum.

[0125] In some embodiments of the present disclosure, image information of each smooth region can be fused based on area weight and brightness weight for subsequent physiological state detection, where the first contribution includes area weight and the second contribution includes brightness weight. Here, the associated fusion process, i.e., the above step S208, can be realized by the following process:

[0126] In step 1, a weighted sum value corresponding to each smooth region is determined based on the product of the area weight and the luminance weight corresponding to each smooth region.

[0127] In step 2, perform weighted fusion on the image information of at least one smooth region according to the corresponding weighted sum value to determine fused image information.

[0128] Here, a weighted sum value corresponding to each smooth region can be determined first, and then, after the image information of each smooth region is assigned a corresponding weighted sum value, the fusion image information can be determined by the weighted sum.

[0129] In some embodiments of the present disclosure, for smooth regions with larger area weights and larger brightness weights, the corresponding weighted sum value is larger, and to some extent, can provide more information support for the fusion image information. For smooth regions with larger area weights and smaller brightness weights, or with smaller area weights and larger brightness weights, the corresponding weighted sum value may be larger or smaller, and needs to be determined based on the image analysis results. The fusion image information obtained by weighted fusion can dig out the effective pixel points on the surface to the maximum extent, and provide more data support for subsequent physiological state detection, which is favorable to improving the detection accuracy.

[0130] In some embodiments of the present disclosure, when the first and second contributions of each smooth region to the physiological state detection result are determined and the fusion image information includes a fused time-domain luminance signal, the physiological state detection method provided in the embodiments of the present disclosure can first determine a time-domain luminance signal corresponding to each smooth region through time-domain processing, and then perform fusion on each time-domain luminance signal according to the first and second contributions of each smooth region, which is convenient for extracting physiological state information based on the fused time-domain luminance signal.

[0131] In some embodiments, in order to improve the accuracy of physiological state detection, frequency domain processing can be performed on the fused time domain luminance signal. More useful information can be analyzed based on the frequency domain signal obtained by frequency domain processing. For example, the width value distribution and energy distribution of each frequency component can be determined, and the frequency value of the key width and energy distribution can be obtained. Here, the physiological state value of the target object can be determined based on the peak value of the frequency domain signal.

[0132] In the embodiment of the present disclosure, when the fusion image information is determined, the time domain luminance signal corresponding to each color channel can be determined according to the luminance value corresponding to the three color channels of the fusion image information. After performing the principal component analysis for each color channel, the physiological state detection for the target object can be realized. That is, the above step S208 can be further realized by the following steps:

[0133] In step 1, for each smooth region, time domain processing is performed on the luminance values ​​of the smooth region in multiple frames of images to determine a time domain luminance signal corresponding to the smooth region.

[0134] In step 2, the time-domain luminance signals corresponding to the plurality of smooth regions are fused according to the first contribution and the second contribution to obtain a fused time-domain luminance signal.

[0135] Here, step S209 provided in the above embodiment can be realized by the following process.

[0136] The fused time domain luminance signal is subjected to frequency domain processing, and a physiological state result is determined based on a peak value of the frequency domain signal obtained by the frequency domain processing.

[0137] In some embodiments, performing frequency domain processing on the fused time domain luminance signal may include the following steps: performing principal component analysis on the fused time domain luminance signal to obtain a corresponding time domain signal, performing frequency domain transformation on the time domain signal to obtain a frequency domain signal representing the physiological state of the target object, and finally determining the physiological state result of the target object based on the peak value of the frequency domain signal.

[0138] In some embodiments of the present disclosure, when a video stream is acquired, a face image of the target object can be extracted from the video stream first. Then, the body archive data of the target object is searched based on the face image, and at the same time, time domain processing and frequency domain processing are performed based on the image information of the face image to obtain an estimated heart rate of the target object. Finally, an estimation is performed on the blood pressure value of the target object based on the found body archive data and the processed estimated heart rate value to obtain a blood pressure measurement result. In the related art, a dedicated device or instrument is required for measurement, which is inconvenient for measurement. Compared with the problems in the related art, the physiological state detection method provided in the embodiments of the present disclosure can determine a heart rate estimate representing the blood pressure fluctuation situation, thereby combining the body archive data to realize accurate measurement of the blood pressure value, and the measurement result is more accurate. Furthermore, there is no need to use a dedicated device, and the measurement can be performed in real time anytime and anywhere, which is practical. Here, blood pressure (BP: Blood Pressure) is the lateral pressure acting on a unit area of ​​the vascular wall when blood flows into a blood vessel. It is the force that causes blood to flow into a blood vessel, and is a very important physiological index that reflects the health condition of the human body in daily life.

[0139] In practical application, the heart pumps blood every time it contracts, and the blood flows through the whole body along the aorta, and the pumped blood is the measured blood pressure. Usually, high and low blood pressure are related to the force of the heart pumping blood, that is, there is a certain relationship between the heartbeat situation of the human body and the blood pressure of the human body. Considering the above, and based on this, here, the image change information corresponding to the face images of multiple frames in a continuous video stream for a certain time is adopted to determine the heart rate estimation value, so that the blood pressure measurement result can be determined by combining the body archive data. The blood pressure value measured in this way can be combined with the heartbeat situation and the specific attributes of different human bodies, so as to be more in line with the needs of actual scenes.

[0140] In some embodiments of the present disclosure, when the target object's physiological state detection result includes a blood pressure measurement result, the above embodiment can perform the following steps S210 to S212. As shown in Fig. 2E, Fig. 2E is a flowchart of a physiological state detection method according to a fifth aspect provided in an embodiment of the present disclosure.

[0141] In step S210, based on the face image, body archive data matching the target object is searched from a predetermined archive library.

[0142] In step S211, time domain processing and frequency domain processing are performed based on image information of a plurality of frames of face images to obtain an estimated heart rate of a target object.

[0143] In step S212, an estimation is made for the blood pressure value of the target subject based on the physical archive data and the estimated heart rate value to obtain a blood pressure measurement result.

[0144] In the embodiment of the present disclosure, the body archive data can include height, weight, gender data, age, etc. This takes into consideration that the body function status of human bodies with different heights, weights, gender data, and ages is different, for example, the heart rate of a child is usually faster than that of an elderly person. Here, the body archive data and the estimated heart rate are combined to estimate the blood pressure value, in order to realize blood pressure measurement applicable to each type of human body, and the applicability is higher.

[0145] In some embodiments of the present disclosure, the above step S210 can be realized in at least one of the following ways.

[0146] In method 1, based on the facial features of a target object extracted from a facial image, body archive data corresponding to the facial features of the target object is searched from a predetermined archive library.

[0147] In method 2, based on the landmark of the target object identified from the face image, the body archive data corresponding to the landmark is searched from a predetermined archive library.

[0148] In the embodiment of the present disclosure, based on the face image, body archive data matching with the target object can be searched from a predetermined archive library. That is, the body archive data of each object can be stored in a predetermined archive library in advance, and when the face image is acquired, the body archive data corresponding to the target object can be searched from the predetermined archive library based on the face image or the face features extracted from the face image, which is easy to operate. Alternatively, some body archive data, such as age, gender, etc., can be obtained by performing an analysis on the face image of the target object.

[0149] In some embodiments of the present disclosure, facial features of a target object can be extracted based on a facial image, and body archive data corresponding to the facial features of the target object can be searched from a predetermined archive library. Here, the facial features can be related features including the distance between the eyes, the distance between the eyebrows and the eyes, etc., or can be feature representations obtained by performing feature extraction on the face image by a deep learning model such as a convolutional neural network for identifying faces. When the facial features of each object are pre-recorded in a predetermined archive library, the body archive data corresponding to the target object can be found based on the similarity between the extracted facial features and each facial feature in the predetermined archive library.

[0150] In some embodiments of the present disclosure, a target object can be identified based on a face image, thereby determining the mark of the target object, and then searching for body archive data corresponding to the mark from a predefined archive library to become the body archive data matching with the target object. That is, different marks can be pre-set for each object, and when performing face comparison based on a face image, the target object and its body archive data corresponding to the mark can be determined.

[0151] In some embodiments of the present disclosure, a heart rate estimate can be obtained by performing time domain processing and frequency domain processing based on the extracted image information of multiple frames of face images. In practical application, a heart rate estimate that more meets actual needs can be obtained based on the time domain processing and frequency domain processing of the image information of multiple color channels. The realization process can refer to the steps of extracting physiological state information based on the image information value of the smooth region to obtain a physiological state detection result of a target object. Here, the physiological state detection result is a heart rate estimate.

[0152] In some embodiments of the present disclosure, the process of achieving blood pressure measurement results for a target object, i.e., the above step S212, can be achieved by the following steps.

[0153] In step 1, a blood shot time corresponding to a heart rate estimate is determined based on a linear fit relationship between the blood shot time and the heart rate. A body surface area corresponding to a target subject is calculated based on gender data, height and weight included in the body archive data. A cardiac elasticity value of the target subject is determined based on weight, age and the heart rate estimate of the target subject included in the body archive data.

[0154] In step 2, a stroke volume corresponding to the target subject is determined based on the blood shot time, body surface area, age contained in the body archive data, and heart rate estimates.

[0155] In step 3, a pulse pressure estimate corresponding to the target object is calculated based on the ratio between the stroke volume and cardiac elasticity corresponding to the target object.

[0156] In step 4, systolic and diastolic pressure values ​​of the target subject are calculated based on the pulse pressure estimate corresponding to the target subject and the gender data contained in the physical archive data.

[0157] In some embodiments, the subject's respective body archive data and the extrapolated heart rate estimate can be combined to realize a blood pressure measurement value including a systolic pressure value and a diastolic pressure value. To facilitate understanding of the above blood pressure measurement process, the following are related terminology in the blood pressure measurement result:

[0158] Systolic pressure: When the heart contracts and pumps blood from the ventricles into the arteries, the pressure that the blood exerts against the arterial walls is the systolic pressure.

[0159] Diastolic pressure: When the heart expands, the pressure exerted by the arterial walls, which have a certain degree of elasticity, on the blood flow is called the diastolic pressure.

[0160] Pulse Pressure: The difference between systolic and diastolic pressure.

[0161] Mean Arterial Pressure: The average arterial blood pressure over one heartbeat period.

[0162] Stroke volume: Stroke volume (SV) is the amount of blood ejected from one ventricle during one heartbeat.

[0163] Ejection Time: Left Ventricular Ejection Time (LVET) is defined as the time interval between the opening and closing of the aortic valve, the systole during which the left ventricle ejects blood into the aorta, and may be abbreviated as (ET).

[0164] For body surface area (BSA), the Stevenson formula is a body surface area formula that is consistent with human body characteristics.

[0165] Cardiac Output (CO): describes the amount of blood the heart pumps per unit time.

[0166] Systemic Vascular Resistance (SVR).

[0167] In some embodiments, specific formulas can be combined and described. For example, the blood shot time corresponding to the estimated heart rate is determined based on the linear fit relationship between the blood shot time and the heart rate, where the linear fit relationship is ET=364.5-1.23*HR, where ET is for representing the blood shot time, HR is for representing the estimated heart rate, and parameters such as 364.5 and 1.23 can be the result of linear fitting verified based on a specific data set.

[0168] In some embodiments, the body surface area corresponding to the target subject can be calculated based on the gender data, height and weight included in the archive data, where the body surface area BSA is calculated based on the health archive information (height, weight) of the target subject and the Stevenson formula.

[0169] In some embodiments, the body surface area of ​​the target subject can be calculated based on BSA=0.0057*Height+0.0121*Weight+0.0882 if the target subject is male, and the body surface area of ​​the target subject can be calculated based on BSA=0.0073*Height+0.0127*Weight-0.2106 if the target subject is female.

[0170] When ET, HR, and BSA are determined, the stroke volume SV corresponding to the target subject can be determined by combining the age Age contained in the physical archive data. It can be determined based on the following formula (1).

[0171] SV=-6.6+0.25*(ET-35)-0.62*HR+40.4*BSA-0.51*Age Formula (1) Based on the ratio of the SV and cardiac elasticity (CE) corresponding to the target, the pulse pressure estimate (Pp) corresponding to the target object can be calculated and obtained by the following formula (2).

[0172] Pp=SV / CE Formula (2) Here, CE=0.013*Weight-0.007*Age-0.004*HR+1.307, and calculate Pm based on mean arterial pressure Pm=CO*SVR, where CO is cardiac output and SVR is systemic vascular resistance. For CO, the average value for men is 5 L / min and for women is 4.5 L / min, and the experienced value for SVR is 18.31 mmhgmin / L.

[0173] Once the mean arterial pressure is determined, the systolic pressure (Ps = Pm + 2 / 3Pp) and the diastolic pressure (Pd = Pm - 1 / 3Pp) can be calculated. The derivation of these two equations is shown in Equation (3).

[0174] Pm=Pd+1 / 3(Ps-Pd) Formula (3) Furthermore, by combining the definition of Pp, Pp = Ps - Pd, we can derive the formula for calculating Ps and Pd.

[0175] In order to better monitor the blood pressure status of the target object, the blood pressure measurement results and the collection times corresponding to the images of multiple frames can be stored in the blood pressure measurement record of the target object, which can store the blood pressure measurement status of the target object in each time period, and can generate a corresponding blood pressure measurement report based on the blood pressure measurement record in a certain pre-set time length, which is convenient for real-time monitoring of the target object. Particularly, in the process of driving a car, real-time blood pressure monitoring can reduce the phenomenon of dangerous driving behavior caused by too high or too low blood pressure, and improve driving safety.

[0176] The related blood pressure measurement report may be, for example, a result of smoothing several measurement results within 5 minutes, or a report of blood pressure dynamic changes within a certain period of time (for example, comparison of blood pressure at different times during the day and change trends). In addition, the related report may be in the context of monitoring blood pressure from other dimensions, and is not specifically limited here.

[0177] In practical application, the blood pressure measurement result can be stored in association with the target object's physical archive data. With the change of various types of physical archive data such as age, weight, etc., the blood pressure measurement result also changes. This is convenient for performing a longer time span physical condition analysis of the target object, and can promptly alert when the physical indicators are abnormal.

[0178] After achieving blood pressure measurement, in the embodiment of the present disclosure, the blood pressure measurement result can further be displayed, thereby providing better in-cabin service to the target subject according to the displayed blood pressure measurement result.

[0179] In some embodiments, in order to improve the accuracy of physiological state detection, the time domain signal can be transformed into a frequency domain, and more useful information can be analyzed based on the transformed frequency domain signal. For example, the width value distribution and energy distribution of each frequency component can be determined, thereby obtaining the frequency value of the key width and energy distribution. The physiological state value of the target object can be determined based on the peak value of the frequency domain signal.

[0180] Take heart rate detection as an example, here, the peak value pmax of the frequency domain signal is determined, and the original heart rate measurement value can be obtained by summing pmax and the heart rate reference value, where pmax represents the heart rate variation, and the heart rate reference value can be determined as the lower limit of the heart rate estimation range based on experience. Furthermore, the heart rate reference value can be adjusted taking into account the influence of factors such as the frame rate of the video, the length of the frequency domain signal, etc.

[0181] After determining the heart rate, the relevant physiological indexes such as blood oxygen saturation and heart rate variability can be measured. For blood oxygen saturation, the time domain signals of HbO2 and Hb are detected by using red light (600-800 nm) and near-infrared light (800-1000 nm), and the corresponding ratio is calculated to obtain blood oxygen saturation. For heart rate variability, after extracting the time domain signal, the intervals between two adjacent peaks are calculated, and multiple interval times are obtained by combining the frame rate, and the standard deviation of these interval times (SDNN: Standard Deviation of NN Intervals) is taken to obtain the heart rate variability.

[0182] Respiratory frequency detection is similar to the method of heart rate detection, with the difference being that the range of respiratory frequency and the range of heart rate are different, and the corresponding reference values ​​are set differently. Respiratory frequency detection can be realized based on the same method as above.

[0183] The embodiment of the present disclosure realizes physiological state detection of images of multiple frames, that is, the image change information corresponding to the images of multiple frames can represent the change situation of the physiological state. In practical application, the physiological state detection result determined for the associated video stream can be updated as the image frames are collected.

[0184] In some embodiments of the present disclosure, when an image of one or more frames included in a new video stream is acquired, the following steps may be performed: Perform face detection on the image in the new video stream, extract face images of target objects in the vehicle cabin, and determine at least one smooth region in the face image of each frame; Perform weighting or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding a physiological state; Update the physiological state detection result based on the information to be extracted; If the detection time length is less than the preset detection time length, perform updating again based on the acquired new video stream until the detection time length reaches the preset detection time length, thereby obtaining an updated physiological state detection result.

[0185] Here, when an image of one or more frames included in the new video stream is acquired, the following steps can be performed: Perform surface detection on the image in the new video stream, extract a surface image of a target object in the vehicle cabin, and determine at least one smooth region in the surface image; From the at least one smooth region, determine at least one region of interest whose image brightness satisfies a predetermined condition; In this way, the physiological state detection result can be updated based on the image information of the at least one region of interest; If the detection time length is less than the preset detection time length, update is performed again based on the acquired new video stream until the detection time length reaches the preset detection time length, thereby obtaining an updated physiological state detection result.

[0186] In some embodiments, when an image of one or more frames included in a new video stream is acquired, the following steps may be performed: Perform face detection on the image in the new video stream to extract a face image of a target object in the vehicle cabin, and determine at least one smooth region in the face image; For each smooth region, generate a time-domain luminance signal corresponding to the smooth region based on pixel luminance information of at least one color channel of the smooth region in the face images of the multiple frames; Perform weighting on the time-domain luminance signal corresponding to the at least one smooth region based on the area of ​​each smooth region to obtain an area-weighted time-domain luminance signal; Update the physiological state detection result based on the area-weighted time-domain luminance signal; If the detection time length is less than the preset detection time length, update again based on the acquired new video stream until the detection time length reaches the preset detection time length, thereby obtaining an updated physiological state detection result.

[0187] In some embodiments, when an image of one or more frames included in a new video stream is acquired, the following steps may be performed: perform face detection on the image in the new video stream to extract a face image of a target object in the vehicle cabin, and determine at least one smooth region in the face image; when a first contribution and a second contribution of each smooth region to a physiological state detection result are determined, perform fusion on image information of the at least one smooth region based on the first contribution and the second contribution to obtain fused image information; perform an update on the physiological state detection result based on the fused image information; if the detection time length is less than the preset detection time length, perform an update again based on the acquired new video stream until the detection time length reaches the preset detection time length, thereby obtaining an updated physiological state detection result.

[0188] Here, heart rate detection is still taken as an example. If it is determined that the preset detection time length is 30s, the video stream can be continuously acquired within 30s. The heart rate measurement value is calculated based on the images of multiple frames of the first video stream (for example, the video stream within 5 seconds from the start), which is still within 30 seconds. In this case, as the image frames are collected, the number of image frames increases, and one new heart rate measurement value can be calculated every time one frame or n frames are added. Smoothing is performed by sliding average, and the measurement is terminated when it reaches 30s, and the final measurement result is obtained.

[0189] For example, in a vehicle environment, a detection progress presentation signal can be generated to present a required detection time length to a target object based on the time length of an acquired video stream and a preset detection time length in one physiological state detection process, so that the target object can measure the physiological state more quickly. For example, when the time length of the acquired video stream (i.e., the detection time during which the current target object's physiological state detection has been continued) is 25 seconds and the preset detection time length is 30 seconds, a voice or screen presentation such as "Please stay still, detection will be completed in 5 seconds" can be presented, or when the current target object's physiological state detection time length is 30 seconds, a voice or screen presentation of "measurement completed" can be presented.

[0190] After achieving physiological state detection, in an embodiment of the present disclosure, the physiological state detection result is further displayed, so that better in-vehicle services can be provided to the target subject through the displayed physiological state detection result.

[0191] In an embodiment of the present disclosure, the physiological state detection result of the target object can be sent to a display in the vehicle cabin, and thus displayed on the display. In this way, the vehicle cabin personnel can monitor their own physiological state status in real time, and if their own physiological state is abnormal, they can promptly consult a doctor or take other necessary measures. At the same time, the physiological state detection result of the target object can also be sent to a server of the physiological state detection application, and thus, when the target object requests to obtain the detection result through the physiological state detection application, the server will send the physiological state detection result to a terminal device used by the target object.

[0192] That is, the target object's physiological state detection result can be recorded in the server, and the server can further perform statistical analysis on the physiological state detection result. For example, the physiological state statistical result of the past month or week can be determined. In this way, when the target object sends a physiological state detection request, the physiological state detection result, statistical result, etc. can be sent to the target object's terminal device, thereby realizing a more comprehensive physiological state evaluation.

[0193] Here, the above-mentioned physiological state detection application can be a specific application program (APP) for performing physiological state detection. The APP can be used to respond to a request for detection results related to a target object, thereby realizing the display of the results in the APP, which is more practical.

[0194] As can be understood by those skilled in the art, in the above method for implementing the invention, the order of steps described does not mean a strict execution order, nor does it constitute any limitation on the implementation process. The specific execution order of steps should be determined by their functions and possible internal logic.

[0195] Based on the same invention concept, the embodiment of the present disclosure further provides a physiological condition detection device corresponding to the physiological condition detection method. The principle by which the device in the embodiment of the present disclosure solves the problem is similar to the above physiological condition detection method in the embodiment of the present disclosure, so that the implementation of the device can refer to the implementation of the method.

[0196] Referring to Fig. 3, Fig. 3 is a schematic diagram of a physiological state detection device provided in an embodiment of the present disclosure, the device includes: an acquisition unit 301 configured to acquire a video stream collected by a photographing device; an extraction unit 302 configured to extract a face image of multiple frames of a target object from the image of multiple frames in the video stream; a determination unit 303 configured to determine at least one smooth region in the face image; a processing unit 304 configured to perform weighting processing or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding a physiological state; and a detection unit 305 configured to perform physiological state information extraction on the information to be extracted to obtain a physiological state detection result of a target object.

[0197] In some embodiments of the present disclosure, the determination unit 303 is further configured to perform face feature extraction on the face image to obtain positions of a plurality of pre-defined key feature points in the face image, and determine at least one smooth region in the face image corresponding to the plurality of pre-defined key feature points based on the positions of the plurality of pre-defined key feature points.

[0198] In some embodiments of the present disclosure, the region attribute information includes region luminance, and the information to be extracted regarding the physiological state includes image information of at least one region of interest. The processing unit 304 is further configured to determine at least one region of interest whose region luminance satisfies a predetermined condition from the at least one smooth region. The detection unit 305 is further configured to perform physiological state information extraction on the image information of the at least one region of interest to obtain a physiological state detection result.

[0199] In some embodiments of the present disclosure, the processing unit 304 is further configured to determine an image brightness threshold corresponding to the surface image and a region image brightness of each smooth region, filter the at least one smooth region based on the image brightness threshold and the region image brightness to obtain at least one candidate smooth region, and select at least one region of interest from the at least one candidate smooth region, whose region brightness satisfies a preset brightness condition.

[0200] In some embodiments of the present disclosure, the image information of the region of interest includes pixel luminance information of three different color channels, and the detection unit 305 is further configured to determine a time domain signal representing a physiological state of the target object according to the pixel luminance information of the three color channels of the region of interest, perform a frequency domain transform on the time domain signal to obtain a frequency domain signal representing the physiological state of the target object, and determine a physiological state detection result of the target object according to a peak value of the frequency domain signal.

[0201] In some embodiments of the present disclosure, the detection unit 305 is further configured to determine, based on the pixel luminance information of the three color channels corresponding to the region of interest, a time-domain luminance signal corresponding to each color channel of the region of interest, and perform principal component analysis on the time-domain luminance signals corresponding to the three different color channels of the region of interest to obtain a time-domain signal.

[0202] In some embodiments of the present disclosure, the region attribute information includes a region area, and the information to be extracted regarding the physiological state includes a weighted time-domain luminance signal. The processing unit 304 is further configured to generate a time-domain luminance signal corresponding to each smooth region based on pixel luminance information of at least one color channel of the smooth region in the surface images of multiple frames, and perform a weighting process on the time-domain luminance signal corresponding to the at least one smooth region based on the area of ​​each smooth region to obtain an area-weighted time-domain luminance signal. The detection unit 305 is further configured to extract physiological state information based on the area-weighted time-domain luminance signal to obtain a physiological state detection result.

[0203] In some embodiments of the present disclosure, the smooth region corresponds to pixel luminance information of three different color channels. The processing unit 304 is further configured to determine a time domain luminance signal corresponding to each color channel of the smooth region based on pixel luminance information corresponding to the three color channels of the smooth region in the surface images of the multiple frames, and for each color channel, perform weighting processing on the time domain luminance signal in the color channel of at least one smooth region based on the area of ​​each smooth region to obtain an area-weighted time domain luminance signal in the color channel. The detection unit 305 is further configured to perform principal component analysis on the area-weighted time domain luminance signals in the multiple different color channels of the smooth region to obtain a time domain signal representing the physiological state of the target object, perform frequency domain processing on the time domain signal representing the physiological state of the target object, and determine a physiological state detection result based on a peak value of the frequency domain signal obtained by the frequency domain processing.

[0204] In some embodiments of the present disclosure, the region attribute information includes a region area and a region brightness, and the information to be extracted regarding the physiological state includes fusion image information. The processing unit 304 is further configured to determine a first contribution of each smooth region to the physiological state detection result based on the area of ​​each smooth region, determine a second contribution of each smooth region to the physiological state detection result based on pixel brightness information of each smooth region, and perform fusion on image information of at least one smooth region based on the first contribution and the second contribution to obtain fusion image information. The detection unit 305 is further configured to perform physiological state information extraction on the fusion image information to obtain a physiological state detection result.

[0205] In some embodiments of the present disclosure, the first contribution includes an area weight, and the second contribution includes a brightness weight. The processing unit 304 is further configured to determine a weighted sum value corresponding to each of the smooth regions based on a product of the area weight and the brightness weight corresponding to each of the smooth regions, and perform weighted fusion on the image information of the at least one smooth region based on the corresponding weighted sum value to determine fused image information.

[0206] In some embodiments of the present disclosure, the fusion image information includes a fused time-domain luminance signal. The processing unit 304 is further configured to, for each smooth region, perform time-domain processing on the luminance values ​​of the smooth region in the images of the multiple frames to determine a time-domain luminance signal corresponding to the smooth region, and perform fusion on the time-domain luminance signals corresponding to the multiple smooth regions based on the first contribution and the second contribution to obtain a fused time-domain luminance signal. The detection unit 305 is configured to perform frequency-domain processing on the fused time-domain luminance signal, and determine a physiological condition result based on a peak value of the frequency-domain signal obtained by the frequency-domain processing.

[0207] In some embodiments of the present disclosure, the processing unit 304 is further configured to search for body archive data matching the target object from a predetermined archive library based on the face image. The detection unit 305 is further configured to perform time domain processing and frequency domain processing based on the image information of the face images of multiple frames to obtain a heart rate estimate of the target object, and perform an estimation on a blood pressure value of the target object based on the body archive data and the heart rate estimate to obtain a blood pressure measurement result.

[0208] In some embodiments of the present disclosure, the processing unit 304 is further configured to search for body archive data corresponding to the face features of the target object from a predetermined archive library based on the face features of the target object extracted from the face image, and to search for body archive data corresponding to the signs from a predetermined archive library based on the signs of the target object identified from the face image.

[0209] In some embodiments of the present disclosure, the detection unit 305 is further configured to: determine a blood shot time corresponding to the heart rate estimate based on a linear fit relationship between the blood shot time and the heart rate; calculate a body surface area corresponding to the target target based on the gender data, height and weight included in the body archive data; determine a cardiac elasticity value of the target target based on the weight, age, and heart rate estimate of the target target included in the body archive data; determine a stroke volume corresponding to the target target based on the blood shot time, body surface area, age, and heart rate estimate included in the body archive data; calculate and obtain a pulse pressure estimate corresponding to the target target based on the ratio between the stroke volume and the cardiac elasticity value corresponding to the target target; and calculate and obtain a systolic pressure value and a diastolic pressure value of the target target based on the pulse pressure estimate corresponding to the target target and gender data included in the body archive data.

[0210] In some embodiments of the present disclosure, the extractor 302 is further configured to determine a surface image of a target object from the detected surface images based on the surface detection result of the video stream and a predetermined riding position, where the predetermined riding position is for indicating a position of the target object to be measured.

[0211] In some embodiments of the present disclosure, the device further includes a display unit configured to transmit the physiological state detection result of the target subject to a display in the vehicle cabin for displaying on the display, and a transmitting unit configured to transmit the physiological state detection result of the target subject to a server of the physiological state measurement application, so that when the target subject requests to obtain the detection result through the physiological state application, the server transmits the physiological state detection result to a terminal device used by the target subject.

[0212] In some embodiments of the present disclosure, the detection unit 304 is further configured to, when a new video stream is acquired, repeatedly execute the steps of extracting multiple frames of face images of the target object from the new video stream until a preset detection time length is reached, determining at least one smooth region in the face image of each frame, performing weighting processing or region selection on the at least one smooth region based on region attribute information, obtaining information regarding the physiological state to be extracted, and updating the physiological state detection result based on the information to be extracted, so as to obtain an updated physiological state detection result.

[0213] In some embodiments of the present disclosure, the device includes a prompting unit configured to generate a detection progress indication signal based on a duration of the captured video stream and a preset detection time length, the detection progress indication signal being for indicating a required detection time length for the target object.

[0214] For the description of the processing process of each part in the device and the interaction process between each part, reference can be made to the relevant description in the above method embodiment, and will not be described in detail here.

[0215] The embodiment of the present disclosure further provides an electronic device. Referring to FIG. 4, FIG. 4 is a structural schematic diagram of the electronic device provided in the embodiment of the present disclosure. The electronic device includes a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions executable by the processor 401 (for example, execution instructions corresponding to the acquisition unit 301, the extraction unit 302, the decision unit 303, the processing unit 304, and the detection unit 305 in the device of FIG. 3). When the electronic device is operating, the processor 401 and the memory 402 communicate with each other through the bus 403, and when the processor 401 executes the machine-readable instructions, the electronic device performs the following processes:

[0216] Acquire the video stream collected by the imaging device; Extracting a multi-frame face image of a target object from the multi-frame images of the video stream; determining at least one smooth region in the surface image; Perform weighting or region selection on at least one smooth region based on the region attribute information to obtain information to be extracted regarding the physiological state; Physiological state information is extracted based on the extraction waiting information, and a physiological state detection result of the target subject is obtained.

[0217] An embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, which, when executed, causes a processor to perform the physiological condition detection method described in the above method embodiment, where the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0218] An embodiment of the present disclosure further provides a computer program product carrying program code, the instructions contained in the program code being capable of performing the physiological condition detection method as described in the method embodiment above.

[0219] An embodiment of the present disclosure further provides a computer program comprising computer readable code, which, when executed by an electronic device, causes a processor of the electronic device to implement the physiological condition detection method described in the above method embodiment.

[0220] Here, the above computer program product can be specifically realized in the form of hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK).

[0221] It is obvious to those skilled in the art that, for convenience and conciseness of description, the specific operation steps of the system and device described above can refer to the corresponding steps in the above method embodiments. In some embodiments provided in the present disclosure, it is understandable that the disclosed system, device and method can be realized in other ways. The above-described device embodiments are merely exemplary. For example, the division of the above units is merely a division of logic functions, and there may be other division methods when actually realized. Furthermore, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the mutual coupling, direct coupling or communication connection between each component shown or described may be an indirect coupling or communication connection through some communication interfaces, devices or units, and may be in an electrical, mechanical or other form.

[0222] The units described as separate elements may or may not be physically separated, and the elements shown as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units, among which some or all units may be selected to realize the objective of the solution of the present embodiment based on the actual needs.

[0223] In addition, all of the functional units in each embodiment of the present disclosure may be integrated into a single processing unit, each unit may exist physically alone, or at least two units may be integrated into a single unit.

[0224] When the above functions are realized in the form of a software functional unit and sold or used as an independent product, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, an essential part of the technical solution of the present disclosure, or a part of the contribution to the prior art, or a part of the technical solution, can be embodied in the form of a software product, and the computer software product includes several instructions stored in one storage medium and causing one electronic device (which may be a PC, a server, or a network device, etc.) to execute all or part of the steps of the method according to each embodiment of the present disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0225] Finally, it should be explained that the above-described embodiments are merely specific implementation modes of the present disclosure, and are intended to illustrate the technical solutions of the present disclosure, and do not limit the present disclosure. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above embodiments, it is understood by those skilled in the art that, within the technical scope disclosed in the present disclosure, those skilled in the art can make modifications or easily conceivable variations to the technical solutions described in the above embodiments, or make equivalent substitutions for some of the technical features thereof. These modifications, variations or substitutions should be included in the scope of protection of the present disclosure without departing from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims. [Industrial Applicability]

[0226] The embodiments of the present disclosure provide a physiological state detection method, an apparatus, an electronic device, a storage medium, and a program, which include the steps of: acquiring a video stream collected by a photographing device, extracting a face image of a target object of multiple frames from the image of multiple frames in the video stream, determining at least one smooth region in the face image, performing weighting or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding the physiological state, and performing physiological state information extraction on the information to be extracted to obtain a physiological state detection result of the target object.

Claims

1. A physiological condition detection method, comprising: acquiring a video stream collected by a capture device; extracting multi-frame face images of a target object from the multi-frame images in the video stream; determining at least one smooth region in the surface image; performing weighting or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding a physiological state; performing physiological state information extraction on the information waiting to be extracted, and obtaining a physiological state detection result of the target object.

2. The step of determining at least one smooth region in the face image comprises: performing face feature extraction on the face image to obtain positions of a plurality of pre-defined key feature points in the face image; and determining at least one smooth region in the face image corresponding to the plurality of preset key feature points based on the locations of the plurality of preset key feature points. The physiological condition detection method according to claim 1 .

3. The region attribute information includes region brightness, and the extraction waiting information includes image information of at least one region of interest. The step of performing weighting processing or region selection on the at least one smooth region based on the region attribute information includes: determining at least one region of interest from the at least one smooth region, the region intensity of which satisfies a predetermined condition; The step of extracting physiological state information from the extraction waiting information and obtaining a physiological state detection result of the target object includes: performing physiological state information extraction on image information of the at least one region of interest to obtain the physiological state detection result; The physiological condition detection method according to claim 1 or 2.

4. The step of determining at least one region of interest, the region luminance of which satisfies a predetermined condition, from the at least one smooth region includes: determining an image brightness threshold corresponding to the surface image and a region image brightness for each smooth region; filtering the at least one smooth region based on the image intensity threshold and the region image intensity to obtain at least one candidate smooth region; selecting the at least one region of interest from the at least one candidate smooth region, the region intensity of which satisfies a preset intensity condition; The physiological condition detection method according to claim 3 .

5. The image information of the region of interest includes pixel brightness information of three different color channels, and the step of extracting physiological state information from the image information of the at least one region of interest to obtain the physiological state detection result includes: determining a time domain signal representative of a physiological state of the target object based on pixel luminance information of the three color channels of the region of interest; performing a frequency domain transformation on the time domain signal to obtain a frequency domain signal representative of the physiological condition of the target subject; and determining a physiological state detection result of the target subject based on a peak value of the frequency domain signal. The physiological condition detection method according to claim 3 or 4.

6. Determining a time domain signal representative of a physiological state of the target object based on pixel luminance information of three color channels of the region of interest, comprising: determining a time domain luminance signal corresponding to each of the three color channels of the region of interest based on pixel luminance information of the three color channels corresponding to the region of interest; performing a principal component analysis on time domain luminance signals corresponding to the three different color channels of the region of interest to obtain the time domain signals. The physiological condition detection method according to claim 5 .

7. The region attribute information includes a region area, and the to-be-extracted information includes a weighted time-domain luminance signal, and the step of performing weighting or region selection on the at least one smooth region based on the region attribute information to obtain information to be extracted regarding a physiological state includes: For each of the smooth regions, generating a time-domain luminance signal corresponding to the smooth region based on pixel luminance information of at least one color channel of the smooth region in the multiple frames of face images; and performing a weighting process on the time-domain luminance signal corresponding to the at least one smooth region based on an area of ​​each of the smooth regions to obtain an area-weighted time-domain luminance signal; The step of extracting physiological state information from the extraction waiting information and obtaining a physiological state detection result of the target object includes: extracting physiological state information based on the area-weighted time-domain luminance signal to obtain the physiological state detection result; The physiological condition detection method according to claim 1 or 2.

8. The smooth regions correspond to pixel luminance information of three different color channels, and for each of the smooth regions, the step of generating a time-domain luminance signal corresponding to the smooth region based on pixel luminance information of at least one color channel of the smooth region in the face images of the multiple frames includes: determining a time domain luminance signal corresponding to each of the color channels of the smooth region based on pixel luminance information corresponding to the three color channels of the smooth region in the face images of the multiple frames; performing a weighting process on the time domain luminance signal corresponding to the at least one smooth region based on an area of ​​each of the smooth regions to obtain an area-weighted time domain luminance signal; For each color channel, performing a weighting process on the time-domain luminance signal in the color channel of the at least one smooth region based on an area of ​​each of the smooth regions to obtain an area-weighted time-domain luminance signal in the color channel; extracting physiological state information based on the area-weighted time-domain luminance signal to obtain the physiological state detection result, performing a principal component analysis on the area-weighted time-domain luminance signals of the smooth region in the different color channels to obtain a time-domain signal representing the physiological state of the target object; performing frequency domain processing on the time domain signal representing the physiological state of the target object; and determining the physiological state detection result based on a peak value of the frequency domain signal obtained by the frequency domain processing. The physiological condition detection method according to claim 7.

9. The region attribute information includes a region area and a region brightness, and the information to be extracted includes fusion image information. The step of performing weighting processing or region selection on the at least one smooth region based on the region attribute information to obtain information to be extracted regarding a physiological state includes: determining a first contribution of each of the smooth regions to the physiological condition detection result based on an area of ​​each of the smooth regions; determining a second contribution of each of the smooth regions to the physiological condition detection result based on pixel luminance information of each of the smooth regions; performing fusion on image information of the at least one smooth region based on the first contribution and the second contribution to obtain the fused image information; The step of extracting physiological state information from the extraction waiting information and obtaining a physiological state detection result of the target object includes: extracting physiological state information from the fusion image information to obtain the physiological state detection result; The physiological condition detection method according to claim 1 or 2.

10. The first contribution includes an area weight, and the second contribution includes a brightness weight, and the step of performing fusion on image information of the at least one smooth region based on the first contribution and the second contribution to obtain the fusion image information includes: determining a weighted sum value corresponding to each of the smooth regions based on a product of an area weight and a luminance weight corresponding to each of the smooth regions; performing weighted fusion on the image information of the at least one smooth region based on the corresponding weighted sum value to determine the fusion image information; The physiological condition detection method according to claim 9.

11. The fusion image information includes a fused time-domain luminance signal, and the step of performing fusion on the image information of the at least one smooth region based on the first contribution and the second contribution to obtain the fusion image information includes: for each smooth region, performing a time domain process on the luminance values ​​of the smooth region in the multiple frames of images to determine a time domain luminance signal corresponding to the smooth region; performing fusion on time-domain luminance signals corresponding to the plurality of smooth regions based on the first contribution and the second contribution to obtain a fused time-domain luminance signal; The step of extracting physiological state information from the fusion image information and obtaining the physiological state detection result includes: performing frequency domain processing on the fused time domain luminance signal; and determining the physiological condition result based on a peak value of the frequency domain signal obtained by the frequency domain processing. The physiological condition detection method according to claim 9.

12. The physiological condition detection method further comprises: searching for matching body archive data of the target object from a predetermined archive library based on the face image; performing time domain processing and frequency domain processing based on image information of the face images of a plurality of frames to obtain an estimated heart rate of the target object; and performing an estimation on a blood pressure value of the target subject based on the physical archive data and the estimated heart rate to obtain a blood pressure measurement result. The physiological condition detection method according to any one of claims 1 to 11.

13. The step of searching for body archive data matching the target object from a predetermined archive library based on the face image includes: retrieving body archive data corresponding to the facial features of the target object from the predetermined archive library based on the facial features of the target object extracted from the facial image; and based on the marker of the target object identified from the face image, searching for body archive data corresponding to the marker from the predetermined archive library. The physiological condition detection method according to claim 12.

14. The step of performing an estimation on a blood pressure value of the target subject based on the physical archive data and the estimated heart rate to obtain a blood pressure measurement result includes: determining a blood shot time corresponding to the estimated heart rate based on a linear fit relationship between the blood shot time and the heart rate; calculating a body surface area corresponding to the target subject based on the gender data, height and weight included in the body archive data; and determining a cardiac elasticity value of the target subject based on the weight, age and the estimated heart rate of the target subject included in the body archive data; determining a stroke volume corresponding to the target subject based on the blood shot time, the body surface area, age included in the body archive data, and the heart rate estimate; calculating a pulse pressure estimate corresponding to the target object based on a ratio between the stroke volume corresponding to the target object and the cardiac elasticity value; and calculating systolic and diastolic pressure values ​​of the target object based on the pulse pressure estimate corresponding to the target object and gender data included in the physical archive data. The physiological condition detection method according to claim 12 or 13.

15. The video stream includes a video stream of a vehicle interior, and the step of extracting a plurality of frame face images of a target object from a plurality of frame images in the video stream includes: determining a surface image of the target object from the detected surface images based on a surface detection result of the video stream and a predetermined riding position, the predetermined riding position being for indicating a position of the target object to be measured; The physiological state detection method according to any one of claims 1 to 14.

16. The physiological condition detection method further comprises: transmitting the physiological state detection result of the target object to a display in the vehicle cabin, thereby displaying it on the display; transmitting the physiological state detection result of the target object to a server of a physiological state measurement application, whereby, when the target object requests to obtain the detection result through the physiological state measurement application, the server transmits the physiological state detection result to a terminal device used by the target object; The physiological condition detection method according to claim 15.

17. The physiological condition detection method further comprises: When a new video stream is acquired, it is detected for a pre-defined detection time period. extracting face images of a plurality of frames of a target object from the new video stream, determining at least one smooth region in the face image of each frame, performing weighting or region selection on the at least one smooth region based on region attribute information, obtaining information on a physiological state to be extracted, and updating the physiological state detection result based on the information to be extracted, repeatedly executing the steps to obtain an updated physiological state detection result. The physiological condition detection method according to any one of claims 1 to 16.

18. The physiological condition detection method further comprises: generating a detection progress indication signal based on a duration of the captured video stream and the preset detection duration, the detection progress indication signal being for indicating a required detection duration for the target object; The physiological condition detection method according to claim 17.

19. A physiological condition detection device, comprising: an acquisition unit configured to acquire a video stream collected by the image capture device; an extractor configured to extract multi-frame face images of a target object from the multi-frame images in the video stream; a determiner configured to determine at least one smooth region in the surface image; A processing unit configured to perform weighting or region selection on the at least one smooth region based on region attribute information to obtain information to be extracted regarding a physiological state; a detection unit configured to perform physiological state information extraction on the extraction waiting information to obtain a physiological state detection result of the target subject.

20. An electronic device including a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is operating, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed, the processor is caused to perform a physiological condition detection method described in any one of claims 1 to 18.

21. A computer readable storage medium storing a computer program, the computer readable storage medium causing a processor to carry out the physiological condition detection method according to any one of claims 1 to 18 when the computer program is executed.

22. A computer program comprising computer readable code, which when executed by an electronic device causes a processor of the electronic device to execute the physiological condition detection method according to any one of claims 1 to 18.

Images