Non-contact physiological feature recognition method, device, equipment and medium

Abstract

The invention discloses a non-contact physiological feature recognition method, device and equipment and a medium. The method comprises the following steps: acquiring a visible light image and an infrared image containing a target person; performing head contour segmentation on the visible light image and the infrared image to obtain a visible light head image and an infrared head image; determining a head contour alignment relationship according to the visible light head image and the infrared head image; performing face key point recognition on the visible light head image to obtain visible light face key points; adopting the head contour alignment relation to transform the visible light face key points to obtain infrared face key points; and determining the physiological features of the target person by adopting the infrared face key points in the infrared head images. According to the embodiment of the invention, the accuracy and robustness of human physiological feature recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of contactless personnel physiological characteristic recognition, and in particular to a contactless physiological characteristic recognition method, device, equipment and medium. Background Art

[0002] The physiological characteristics of people, such as heart rate and respiratory rate, can be used to evaluate the psychological changes of people. Inmates in the supervision facilities are affected by the closed space and limited interpersonal communication, and they are under great psychological pressure and have a high incidence of mental illness. When the heart rate variability of a group increases, violent behavior and group conflicts are more likely to occur. Monitoring the physiological data such as heart rate and respiratory rate of prisoners in the supervision facilities can provide reference data for evaluating the psychological changes of prisoners and preventing management risks of the supervision facilities. Inmates in the supervision facilities are called at least once in the morning and evening every day. When calling the roll, the camera collects images and replaces the manual calling with face recognition, which can improve the efficiency of calling the roll and reduce errors. When calling the roll by face recognition, the camera is integrated and deployed indoors to avoid exposure to sunlight. When calling the roll by face recognition, the head movement of the person is small and does not block the face. The camera can be used for non-contact recognition of the physiological characteristics of the person when calling the roll by face. Summary of the invention

[0003] The present invention provides a non-contact physiological characteristic recognition method, device, equipment and medium to solve the problems that the existing non-contact personnel physiological characteristic recognition methods have a certain impact on the human body, or have a limited scope of application or large errors.

[0004] According to one aspect of the present invention, there is provided a non-contact physiological feature recognition method, comprising:

[0005] Acquire visible light images and infrared images containing the target person;

[0006] Performing head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image;

[0007] Determining a head contour alignment relationship according to the visible light head image and the infrared head image;

[0008] Performing facial key point recognition on the visible light head image to obtain visible light facial key points;

[0009] Using the head contour alignment relationship, transforming the visible light face key points to obtain infrared face key points;

[0010] The infrared face key points in each infrared head image are used to determine the physiological characteristics of the target person.

[0011] According to another aspect of the present invention, there is provided a non-contact physiological feature recognition device, comprising:

[0012] An image acquisition module is used to acquire visible light images and infrared images containing the target person;

[0013] A contour segmentation module, used to perform head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image;

[0014] A contour alignment module, used to determine a head contour alignment relationship according to the visible light head image and the infrared head image;

[0015] A facial key point module is used to perform facial key point recognition on the visible light head image to obtain visible light facial key points; and transform the visible light facial key points to obtain infrared facial key points using the head contour alignment relationship;

[0016] The physiological feature determination module is used to determine the physiological features of the target person using infrared face key points in each infrared head image.

[0017] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the non-contact physiological feature recognition method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0019] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the non-contact physiological feature recognition method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the non-contact physiological feature recognition method described in any embodiment of the present invention when executed.

[0021] The embodiment of the present invention uses an ordinary optical camera to shoot a visible light image of a target person, and uses an infrared thermal imaging camera to shoot an infrared image of the target person. The visible light image and the infrared image are segmented for head contours respectively to obtain a visible light head image and an infrared head image. The head contour alignment relationship is determined according to the visible light head image and the infrared head image. The face key points are identified for the visible light head image to obtain visible light face key points. According to the head contour alignment relationship, the visible light face key points are transformed to obtain infrared face key points. The image misalignment caused by the position difference when the optical camera and the infrared thermal imaging camera are photographed can be calibrated by the head contour alignment relationship, and the infrared face key points are determined according to the visible light face key points. According to the infrared face key points in each infrared head image, the region of interest in the infrared head image is determined, and the physiological characteristics of the target person are determined according to the temperature change of the region of interest in the infrared head image. There is no need to make direct contact with the target person, and the process of determining the physiological characteristics of the target person is simple and quick. Infrared thermal imaging technology can capture the temperature change of the face of the target person in real time, providing the possibility for real-time monitoring of physiological characteristics. The infrared face key points in the infrared head image can be accurately located, so as to accurately determine the area of ​​interest, and can capture the slight temperature changes in different areas of the face, which provides a guarantee for the accurate judgment of physiological characteristics. The visible light head image provides rich facial details and is suitable for facial key point recognition under most lighting conditions. Infrared images are not sensitive to lighting conditions, but can capture the thermal information of objects. By fusing the information of these two modalities, the accuracy and robustness of human physiological feature recognition can be significantly improved.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 is a first flow chart of a non-contact physiological feature recognition method provided by an embodiment of the present invention;

[0025] Figure 2 is a second flow chart of a non-contact physiological feature recognition method provided by an embodiment of the present invention;

[0026] Figure 3It is a structural schematic diagram of a non-contact physiological characteristic recognition device provided by an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1 This is the first flow chart of a non-contact physiological feature recognition method provided by an embodiment of the present invention. This embodiment is applicable to the case of realizing non-contact collection of physiological features of personnel. The method can be executed by a non-contact physiological feature recognition device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with corresponding data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110, obtaining a visible light image and an infrared image containing a target person; performing head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image.

[0032] The visible light image of the target person is captured by an ordinary optical camera, and the infrared image of the target person is captured by an infrared thermal imaging camera. The image segmentation algorithm is used to segment the target person and the background image in the visible light image and the infrared image to obtain the visible light head image and the infrared head image.

[0033] S120: Determine a head contour alignment relationship according to the visible light head image and the infrared head image.

[0034] S130, performing facial key point recognition on the visible light head image to obtain visible light facial key points.

[0035] S140: Using the head contour alignment relationship, transform the visible light face key points to obtain infrared face key points.

[0036] The head contour alignment relationship between the visible light head image and the infrared head image is determined by feature matching, edge detection or morphological processing. The alignment relationship may include transformation parameters such as rotation, scaling and translation. A facial key point detection algorithm, such as ASM (Active Shape Model), AAM (Active Appearance Model) or deep learning models such as convolutional neural networks, is used to identify the facial key points of the visible light head image, and the facial key points in the visible light image are identified as the visible light face key points. These key points include the eyebrows, eyebrow peaks, eyebrow tails, inner eye corners, outer eye corners, eyelid edges, nose tips, nose wings, nose bridges, mouth corners, lip peaks, lip valleys and facial contours. Based on the head contour alignment relationship between the visible light head image and the infrared head image, the visible light face key points are transformed to match the head posture in the infrared head image. The transformed visible light face key points are the infrared face key points, and the infrared face key points in the infrared head image have the same relative position and shape as the visible light face key points in the visible light image.

[0037] The head contour alignment relationship is determined based on the visible light head image and the infrared head image. The visible light face key points are identified by the face key points of the visible light head image. The infrared face key points are transformed by the head contour alignment relationship. The existing face key point recognition algorithm based on the visible light head image and a large number of open source face databases can be fully utilized, and the head contour alignment relationship between the visible light head image and the infrared head image can be determined by combining feature matching, edge detection or morphological processing methods. The image misalignment caused by the position difference when the optical camera and the infrared thermal imaging camera are taken is eliminated by calibrating the head contour alignment relationship, and the infrared face key points are determined according to the visible light face key points. The research and development cost of the algorithm is reduced, and the usability and practicality of the algorithm are improved; the computational complexity in subsequent processing is reduced, and the operation efficiency of the algorithm is improved.

[0038] S150, using infrared face key points in each infrared head image to determine physiological characteristics of the target person.

[0039] When the target person breathes, the nostril area will show color changes in the infrared image due to the temperature changes of the inhaled and exhaled air. The temperature changes in the nostril area can be used to calculate the target person's breathing frequency in a non-contact manner. There are multiple blood vessel dense areas in the human face area. The temperature changes caused by blood circulation in these areas can be used to calculate the target person's heart rate in a non-contact manner. Determine the facial area whose temperature changes can reflect the physiological characteristics of the target person as the area of ​​interest. According to the infrared face key points in each infrared head image, determine the area of ​​interest in the infrared head image, and determine the physiological characteristics of the target person according to the temperature changes in the area of ​​interest in the infrared head image. There is no need to make direct contact with the target person, and the process of determining the physiological characteristics of the target person is simple and fast, which improves work efficiency. Infrared thermal imaging technology can capture the temperature changes of the target person's face in real time, providing the possibility for real-time monitoring of physiological characteristics. The infrared face key points in the infrared head image can be accurately located, so as to accurately determine the area of ​​interest, and can capture the tiny temperature changes in different areas of the face, which provides a guarantee for the accurate judgment of physiological characteristics.

[0040] Among the existing methods for identifying physiological characteristics of people: Ultrasonic Doppler technology uses the Doppler effect formed by the emitted ultrasonic signal hitting the surface of a moving object to form a reflection or scattering, thereby achieving non-invasive and non-invasive detection, and is often used for continuous detection of fetal heart rate.

[0041] Microwave or millimeter wave bio-radar technology takes the human chest as the research object, uses the radar transmitter to generate high-power high-frequency oscillation pulses, converts them into beam-shaped electromagnetic wave signals through the antenna, transmits them to the object to be detected, and receives the echo reflected by the target object, and obtains the heart rate signal by analyzing the changes in the frequency and phase of the echo signal. The use of radar to detect vital signs is not affected by external environmental factors such as weather, temperature and light intensity, and has strong penetrating power. It can penetrate obstacles such as clothing and walls to obtain amplitude changes caused by life movements, so it is mostly used to search for survivors in ruins.

[0042] The imaging-based photoplethysmography technology is developed on the basis of photoplethysmography. Unlike photoplethysmography, the imaging-based photoplethysmography technology can use the visible light camera on the market to image the target object, so it can directly call the camera on mobile devices such as mobile phones and computers or monitoring equipment to obtain video information including the measured parts of the human body, and use a series of algorithms such as Euler signal amplification to extract the color changes of the human face epidermis in the video to obtain heart rate information.

[0043] However, the ultrasound instrument used in ultrasonic Doppler technology is an active detection. In order to accurately measure, the ultrasound instrument needs to be in contact with the surface of the body, and the signal radiation it emits has a certain impact on the human body.

[0044] Microwave or millimeter wave bio-radar technology requires the subject to remain relatively still when collecting information, and is easily affected by other non-target clutter in the background, resulting in large errors. Moreover, radar and ultrasonic Doppler technology are both active devices, and the electromagnetic waves they emit are emitted by antennas. The closer to the antenna and transmitter, the greater the radiation, which has a certain impact on the human body and has a limited scope of application.

[0045] Imaging photoplethysmography technology is susceptible to factors such as light intensity and user motion interference in real scenarios, which may affect the accuracy and robustness of the test results.

[0046] The embodiment of the present invention uses an ordinary optical camera to shoot a visible light image of a target person, and uses an infrared thermal imaging camera to shoot an infrared image of the target person. The visible light image and the infrared image are segmented for head contours respectively to obtain a visible light head image and an infrared head image. The head contour alignment relationship is determined according to the visible light head image and the infrared head image. The face key points are identified for the visible light head image to obtain visible light face key points. According to the head contour alignment relationship, the visible light face key points are transformed to obtain infrared face key points. The image misalignment caused by the position difference when the optical camera and the infrared thermal imaging camera are photographed can be calibrated by the head contour alignment relationship, and the infrared face key points are determined according to the visible light face key points. According to the infrared face key points in each infrared head image, the region of interest in the infrared head image is determined, and the physiological characteristics of the target person are determined according to the temperature change of the region of interest in the infrared head image. There is no need to make direct contact with the target person, and the process of determining the physiological characteristics of the target person is simple and quick. Infrared thermal imaging technology can capture the temperature change of the face of the target person in real time, providing the possibility for real-time monitoring of physiological characteristics. The infrared face key points in the infrared head image can be accurately located, so as to accurately determine the area of ​​interest, and can capture the slight temperature changes in different areas of the face, which provides a guarantee for the accurate judgment of physiological characteristics. The visible light head image provides rich facial details and is suitable for facial key point recognition under most lighting conditions. Infrared images are not sensitive to lighting conditions, but can capture the thermal information of objects. By fusing the information of these two modalities, the accuracy and robustness of human physiological feature recognition can be significantly improved.

[0047] In an optional embodiment, infrared facial key points in each infrared head image are used to determine the physiological characteristics of the target person, including: using the infrared facial key points in each infrared head image to extract the region of interest in each infrared head image; determining a differential image between the regions of interest in adjacent frames, and performing threshold processing on the differential image to obtain a differential result; performing autoregressive frequency spectrum analysis on each differential result to obtain the physiological characteristics of the target person.

[0048] The temperature of blood is different from that of the skin, and the periodic beating of blood vessels will also be accompanied by periodic changes in the grayscale value of the image. The higher the temperature in the image, the higher the grayscale value, and the closer the color is to white. According to the infrared face key points in each infrared head image, the region of interest in the infrared head image is determined, and the physiological characteristics of the target person are determined according to the temperature changes of the region of interest in the infrared head image.

[0049] Specifically, since most pixels in the region of interest do not change over time, these pixels may increase the noise of the time series signal of the region of interest, so the frame difference method is used to remove the noise in the region of interest. For the continuous K+1 frames of the region of interest, the two adjacent frames are subtracted pixel by pixel to obtain the difference image between the adjacent frames. The difference image shows the change of pixel values ​​in the region of interest from the previous frame to the current frame. The threshold is set to convert the difference image into a binary image. When the pixel value in the difference image is greater than or equal to the threshold, it is considered that the pixel point has changed significantly, and the corresponding pixel point is set to 1; when the pixel value in the difference image is less than the threshold, it is considered that the pixel point has not changed, and the corresponding pixel point is set to 0. The number of pixels with a value of 1 in the binary image is counted as the difference result. For the continuous K+1 frames of the region of interest, the difference image between the continuous adjacent frames is determined, and then the continuous binary image is determined. The number of pixels with a value of 1 in each binary image is counted to obtain the time series signal of the region of interest. The time series signal of the region of interest is subjected to autoregressive frequency spectrum analysis, and the physiological characteristics of the target person are extracted by analyzing the specific frequency components in the frequency spectrum. For example, heart rate usually corresponds to a specific frequency peak in the frequency spectrum.

[0050] By calculating the difference image between the regions of interest of adjacent frames, the slight changes of the face can be clearly captured, providing a guarantee for the accurate judgment of physiological characteristics. Threshold processing of the difference image can distinguish the pixels with significant changes from the background noise, thereby simplifying the subsequent data processing and analysis. The infrared face key points in the infrared head image are used to extract the region of interest, and the physiological characteristics of the target person are obtained through differential image, threshold processing and autoregressive frequency spectrum analysis, which has the advantages of improving recognition accuracy and robustness, improving the efficiency and accuracy of physiological feature extraction, and enhancing the diversity and flexibility of application scenarios. Extracting the physiological characteristics of the target person from video data provides an effective means for remote monitoring and analysis.

[0051] Optionally, the regions of interest include the nose area, the left forehead area, the right forehead area, the left cheek area, the right cheek area and the chin area; the autoregressive frequency spectrum analysis is performed on the differential results to obtain the physiological characteristics of the target person, including: performing autoregressive frequency spectrum analysis on the differential results of each nose area to obtain the breathing frequency of the target person; performing autoregressive frequency spectrum analysis on the differential results of each left forehead area, each right forehead area, each left cheek area, each right cheek area and each chin area to obtain the first heart rate, the second heart rate, the third heart rate, the fourth heart rate and the fifth heart rate; taking the first heart rate, the second heart rate, the third heart rate, the fourth heart rate and the fifth heart rate with the highest number of occurrences as the heart rate of the target person.

[0052] Autoregressive spectrum analysis is performed on the differential results of each nose area, and the target person's breathing rate is estimated according to the peak position in the spectrum. Autoregressive spectrum analysis is performed on the differential results of each left forehead area, each right forehead area, each left cheek area, each right cheek area, and each chin area, and the first heart rate, second heart rate, third heart rate, fourth heart rate, and fifth heart rate of the target person are estimated according to the peak position in the spectrum. The heartbeat signal behaves differently in different facial areas, for example, it is affected by many factors, such as skin thickness, blood circulation status, etc., so the heart rate estimation values ​​in different areas may be different. The heart rate value with the most occurrences reflects the common trend of multiple facial areas. The heart rate with the highest occurrence among the five heart rate values ​​is taken as the heart rate of the target person to obtain the most reliable heart rate estimation. By analyzing the data of multiple facial areas, the accuracy and robustness of the heart rate and breathing rate estimation can be improved. The data of different areas can verify each other and reduce the influence of the error of a single area.

[0053] Figure 2 FIG. 2 is a second flow chart of a non-contact physiological feature recognition method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0054] S210, obtaining a visible light image and an infrared image containing a target person; performing head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image.

[0055] S220: Determine a head contour alignment relationship according to the visible light head image and the infrared head image.

[0056] Optionally, determining the head contour alignment relationship based on the visible light head image and the infrared head image includes: extracting first contour key points from the contour of the visible light head image, and extracting second contour key points from the contour of the infrared head image; wherein the first contour key points and the second contour key points include the highest point of the top of the head and ear key points; and using the first contour key points and the second contour key points to calculate an affine transformation relationship as the head contour alignment relationship.

[0057] Affine transformation is a linear transformation that preserves the parallelism and proportional relationship of straight lines and allows rotation, scaling, and translation. The highest point on the top of the head and the key point of the ear are extracted from the contour of the visible light head image as the first contour key point, and the highest point on the top of the head and the key point of the ear are extracted from the contour of the infrared head image as the second contour key point. The key point pair is composed of the first contour key point and the second contour key point, and the affine transformation matrix of the key point pair is calculated to determine the affine transformation relationship. The calculated affine transformation matrix is ​​applied to the infrared head image to align the infrared head image with the visible light head image. Specifically, the coordinates of each pixel of the infrared head image are transformed according to the affine transformation relationship to match the coordinate system of the visible light head image.

[0058] By calculating the affine transformation relationship as the head contour alignment relationship from the key points extracted from the contours of the visible light head image and the infrared head image, the alignment accuracy can be significantly improved, the robustness can be enhanced, the subsequent processing can be simplified, the visual experience can be improved, multimodal image processing can be supported, and automated processing can be facilitated.

[0059] S230: Input the visible light head image into a pre-trained facial key point recognition model to obtain visible light facial key points.

[0060] The facial key point recognition model is trained in the following manner: obtaining a current sample facial image and a current true contour of the current sample facial image from a training set, and randomly selecting one from other true contours of other sample facial images as an initial estimated contour of the current sample facial image; wherein both the current true contour and the initial estimated contour include facial key points; using the initial estimated contour as the current estimated contour, training a regressor according to the current sample facial image, the current estimated contour and the current true contour, and obtaining a facial key point recognition model.

[0061] Specifically, the facial key point recognition model uses a set of regressors, such as regression forest or random forest regressors, combined with a gradient tree boosting algorithm to estimate the location of facial key points. The training process of the facial key point recognition model includes an initialization process and a regressor training process. The training data set of the facial key point recognition model includes facial images and real contours.

[0062] During the initialization process of the facial key point recognition model, an index is randomly selected, and the facial image corresponding to this index is selected from the training data set as the current sample facial image, and the real contour corresponding to this index is selected as the current real contour of the current sample facial image. The current real contour of the current sample facial image in the training data set is excluded, and a real contour is randomly selected from the training data set that excludes the current real contour of the current sample facial image as the initial estimated contour of the current sample facial image. That is, the initial estimated contour of the current sample facial image is randomly selected from the training data set, and the current real contour of the current sample facial image is excluded, so as to prevent the facial key point recognition model from relying on the real contour, thereby improving the generalization ability of the facial key point recognition model for unknown data. The initialization process of the facial key point recognition model is shown in the following formulas (1.1)-(1.3).

[0063] π i ∈{1,...,n}(1.1)

[0064]

[0065] Among them, π i represents a random index; {S1,...,S n} represents the set of true contours in the training dataset; Represents the current true contour of the current sample face image corresponding to the random index; Represents the initial estimated contour of the current sample face image; Represents the difference between the current true contour and the initial estimated contour. is the initial triplet used for target update, is the current sample face image corresponding to the random index. The initial estimated contour in the initialization process is used as the current estimated contour to train the regressor.

[0066] In the regressor training process of the facial key point recognition model, the current regressor is trained by minimizing the square error based on the current sample facial image, the current estimated contour and the current true contour. The training data set is updated for the next regressor in the cascade based on the current estimated contour. Specifically, the difference between the current true contour and the current estimated contour of the current sample facial image in the training data set is updated. The next regressor in the cascade continues to fit the current estimated contour of the current sample facial image based on the updated training data set, and this process is repeated until the training of the regressors in the regressor set of the facial key point recognition model is completed. Each regressor helps to improve the accuracy of contour labeling. The regressor training process of the facial key point recognition model is shown in the following formulas (2.1)-(2.2).

[0067]

[0068] Among them, r t is the tth regressor in the cascade, and its output is the position update of the key points of the current sample facial image; is the current sample facial image corresponding to the random index; represents the current estimated contour of the current sample face image fitted by the t-th regressor; represents the current estimated contour of the current sample face image fitted by the t+1th regressor; Represents the difference between the current true contour and the current estimated contour of the current sample face image fitted by the t+1th regressor.

[0069] Optionally, during the training process, a weight factor is introduced for each training data in the training data set, and these weight factors are considered during the iteration process of the face key point recognition model, thereby improving the accuracy of the face key point recognition model. Furthermore, the face key point recognition model can be tuned using a public data set. For example, the HELEN face data set, which includes 2330 images; the LFPW face data set, which includes 778 images; the LFW face data set, which includes 2825 images; the 300-W (300 faces In-the-wild) data set, which consists of 300 indoor and 300 outdoor images. The face key point recognition model is tuned using the above data sets to improve the generalization ability of the face key point recognition model.

[0070] The facial key point recognition model is determined by combining a regressor set with a gradient tree boosting algorithm, which can efficiently estimate the position of the visible light facial key points. The facial key point recognition model not only has strong learning and generalization capabilities, but can also handle high-dimensional data and complex nonlinear relationships, improving the accuracy and robustness of facial key point positioning.

[0071] S240: Using the head contour alignment relationship, transform the visible light face key points to obtain infrared face key points.

[0072] S250: Using infrared face key points in each infrared head image to determine physiological characteristics of the target person.

[0073] The embodiment of the present invention uses an ordinary optical camera to shoot a visible light image of a target person, and uses an infrared thermal imaging camera to shoot an infrared image of the target person. The visible light image and the infrared image are segmented to obtain a visible light head image and an infrared head image. The head contour alignment relationship is determined according to the visible light head image and the infrared head image, and the face key point recognition model is determined by combining a regressor set and a gradient tree boosting algorithm, so that the position of the visible light face key point can be efficiently estimated. The face key point recognition model not only has strong learning and generalization capabilities, but also can process high-dimensional data and complex nonlinear relationships, and improve the accuracy and robustness of face key point positioning. According to the head contour alignment relationship, the visible light face key points are transformed to obtain the infrared face key points. The head contour alignment relationship calibration eliminates the image dislocation caused by the position difference when the optical camera and the infrared thermal imaging camera are taken. According to the infrared face key points in each infrared head image, the region of interest in the infrared head image is determined, and the physiological characteristics of the target person are determined according to the temperature change of the region of interest in the infrared head image. There is no need to directly contact the target person, and the process of determining the physiological characteristics of the target person is simple and fast. Infrared thermal imaging technology can capture the temperature changes of the target person's face in real time, providing the possibility for real-time monitoring of physiological characteristics. The infrared face key points in the infrared head image can be accurately located, so as to accurately determine the area of ​​interest, and can capture the slight temperature changes in different areas of the face, providing a guarantee for the accurate judgment of physiological characteristics. Visible light head images provide rich facial details and are suitable for facial key point recognition under most lighting conditions; infrared images are not sensitive to lighting conditions, but can capture the temperature information of objects. By fusing the information of these two modalities, the accuracy and robustness of physiological feature recognition can be significantly improved.

[0074] Figure 3 FIG. 1 is a schematic diagram of the structure of a non-contact physiological feature recognition device provided by an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0075] An image acquisition module 310 is used to acquire a visible light image and an infrared image containing a target person;

[0076] A contour segmentation module 320 is used to perform head contour segmentation on the visible light image and the infrared image to obtain a visible light head image and an infrared head image;

[0077] A contour alignment module 330, configured to determine a head contour alignment relationship according to the visible light head image and the infrared head image;

[0078] The face key point module 340 is used to perform face key point recognition on the visible light head image to obtain visible light face key points; and transform the visible light face key points to obtain infrared face key points using the head contour alignment relationship;

[0079] The physiological feature determination module 350 is used to determine the physiological features of the target person using the infrared face key points in each infrared head image.

[0080] The non-contact physiological characteristic recognition device provided by the embodiment of the present invention can execute the non-contact physiological characteristic recognition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0081] Optionally, a contour alignment module is specifically used to extract a first contour key point from the contour of the visible light head image, and to extract a second contour key point from the contour of the infrared head image; wherein the first contour key point and the second contour key point include the highest point of the top of the head and the ear key point; and the first contour key point and the second contour key point are used to calculate the affine transformation relationship as the head contour alignment relationship.

[0082] Optionally, the facial key point module includes a visible light facial key point unit, which is used to input the visible light head image into a pre-trained facial key point recognition model to obtain visible light facial key points;

[0083] The facial key point recognition model is trained in the following manner: obtaining a current sample facial image and a current true contour of the current sample facial image from a training set, and randomly selecting one from other true contours of other sample facial images as the current estimated contour of the current sample facial image; wherein both the current true contour and the current estimated contour include facial key points; taking the initial estimated contour as the current estimated contour, training a regressor according to the current sample facial image, the current estimated contour and the current true contour, and obtaining a facial key point recognition model.

[0084] Optionally, a physiological characteristic determination module includes:

[0085] An area of ​​interest determination unit, used to extract an area of ​​interest in each infrared head image by using infrared face key points in each infrared head image;

[0086] A difference result unit, used to determine a difference image between regions of interest of adjacent frames, and perform threshold processing on the difference image to obtain a difference result;

[0087] The physiological feature determination unit is used to perform autoregressive frequency spectrum analysis on each differential result to obtain the physiological features of the target person.

[0088] Optionally, the regions of interest include a nose area, a left forehead area, a right forehead area, a left cheek area, a right cheek area and a chin area; the physiological feature determination unit is specifically used to: perform autoregressive spectrum analysis on the differential results of each nose area to obtain the breathing frequency of the target person; perform autoregressive spectrum analysis on the differential results of each left forehead area, each right forehead area, each left cheek area, each right cheek area and each chin area to obtain a first heart rate, a second heart rate, a third heart rate, a fourth heart rate and a fifth heart rate; and take the first heart rate, the second heart rate, the third heart rate, the fourth heart rate and the fifth heart rate with the highest number of occurrences as the heart rate of the target person.

[0089] The non-contact physiological characteristic recognition device further described can also execute the non-contact physiological characteristic recognition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0090] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0091] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0092] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0093] The processor 41 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as a non-contact physiological feature recognition method.

[0094] In some embodiments, the contactless physiological feature recognition method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the contactless physiological feature recognition method described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to perform the contactless physiological feature recognition method in any other appropriate manner (for example, by means of firmware).

[0095] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0097] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0099] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A non-contact physiological feature recognition method, characterized in that: include: Acquire visible light images and infrared images containing the target person; Performing head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image; Determine a head contour alignment relationship according to the visible light head image and the infrared head image; Performing facial key point recognition on the visible light head image to obtain visible light facial key points; Using the head contour alignment relationship, transforming the visible light face key points to obtain infrared face key points; The infrared face key points in each infrared head image are used to determine the physiological characteristics of the target person.

2. The method according to claim 1, characterized in that Determining a head contour alignment relationship according to the visible light head image and the infrared head image includes: Extracting a first contour key point from the contour of the visible light head image, and extracting a second contour key point from the contour of the infrared head image; wherein the first contour key point and the second contour key point include the highest point of the top of the head and the key point of the ear; The first contour key points and the second contour key points are used to calculate an affine transformation relationship as the head contour alignment relationship.

3. The method according to claim 1, characterized in that Performing facial key point recognition on the visible light head image to obtain visible light facial key points includes: Inputting the visible light head image into a pre-trained facial key point recognition model to obtain visible light facial key points; The facial key point recognition model is trained by: obtaining a current sample facial image and a current true contour of the current sample facial image from a training set, and randomly selecting one from other true contours of other sample facial images as a current estimated contour of the current sample facial image; wherein both the current true contour and the current estimated contour include facial key points; The initial estimated contour is used as the current estimated contour, and a regressor is trained according to the current sample facial image, the current estimated contour and the current true contour to obtain a facial key point recognition model.

4. The method according to claim 1, characterized in that: The infrared face key points in each infrared head image are used to determine the physiological characteristics of the target person, including: Using infrared face key points in each infrared head image, extracting the region of interest in each infrared head image; Determine a differential image between regions of interest of adjacent frames, and perform threshold processing on the differential image to obtain a differential result; Autoregressive frequency spectrum analysis is performed on each differential result to obtain the physiological characteristics of the target person.

5. The method according to claim 4, characterized in that The regions of interest include a nose region, a left forehead region, a right forehead region, a left cheek region, a right cheek region, and a chin region; The autoregressive frequency spectrum analysis is performed on each difference result to obtain the physiological characteristics of the target person, including: Perform autoregressive spectrum analysis on the differential results of each nose area to obtain the breathing frequency of the target person; Performing autoregressive spectrum analysis on the difference results of each left forehead area, each right forehead area, each left cheek area, each right cheek area, and each chin area, respectively, to obtain a first heart rate, a second heart rate, a third heart rate, a fourth heart rate, and a fifth heart rate; The heart rate with the highest number of occurrences among the first heart rate, the second heart rate, the third heart rate, the fourth heart rate and the fifth heart rate is taken as the heart rate of the target person.

6. A non-contact physiological feature recognition device, characterized in that: The device comprises: An image acquisition module is used to acquire visible light images and infrared images containing the target person; A contour segmentation module, used to perform head contour segmentation on the visible light image and the infrared image respectively to obtain a visible light head image and an infrared head image; A contour alignment module, used to determine a head contour alignment relationship according to the visible light head image and the infrared head image; A facial key point module is used to perform facial key point recognition on the visible light head image to obtain visible light facial key points; and transform the visible light facial key points to obtain infrared facial key points using the head contour alignment relationship; The physiological feature determination module is used to determine the physiological features of the target person using infrared face key points in each infrared head image.

7. The device according to claim 6, characterized in that The contour alignment module is specifically used for: Extracting a first contour key point from the contour of the visible light head image, and extracting a second contour key point from the contour of the infrared head image; wherein the first contour key point and the second contour key point include the highest point of the top of the head and the key point of the ear; The first contour key points and the second contour key points are used to calculate an affine transformation relationship as the head contour alignment relationship.

8. The device according to claim 6, characterized in that The facial key point module includes a visible light facial key point unit, which is used to input the visible light head image into a pre-trained facial key point recognition model to obtain visible light facial key points; The facial key point recognition model is trained by: obtaining a current sample facial image and a current true contour of the current sample facial image from a training set, and randomly selecting one from other true contours of other sample facial images as a current estimated contour of the current sample facial image; wherein both the current true contour and the current estimated contour include facial key points; The initial estimated contour is used as the current estimated contour, and a regressor is trained according to the current sample facial image, the current estimated contour and the current true contour to obtain a facial key point recognition model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the contactless physiological feature recognition method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the non-contact physiological feature recognition method according to any one of claims 1 to 5 when executed.

Images