Forehead position correction method and device combined with depth calculation, equipment and medium

Abstract

The invention relates to the technical field of intelligent nursing, solves the problems that the forehead position cannot be accurately corrected and the tracking error of the forehead area is large in the prior art, and provides a forehead position correction method and device combined with depth calculation, equipment and a medium. The method comprises the following steps: performing face detection and face tracking on a real-time image, and determining reference point position information through nose and mouth key point position information; determining reference point depth information according to the nose and mouth key point depth information; according to the first reference point position information and the first reference point depth information in the first image, combining the second reference point position information and the second reference point depth information in the second image, and determining the displacement and the depth variation of the reference point; and according to the displacement and the depth variation, judging the face state, and when it is judged that the face deforms, performing correction to determine the position information of the target forehead. According to the invention, the forehead area positioning error is reduced.

Description

[0001] This application is a divisional application of the invention patent application filed on July 3, 2024, with the invention name “Heart rate measurement stabilization method, device and equipment based on facial image processing” and application number 202410881480.0. Technical Field

[0002] The present invention relates to the field of intelligent nursing technology, and in particular to a forehead position correction method, device, equipment and medium combined with depth calculation. Background Art

[0003] Correcting the forehead area of ​​a face is an important part of computer vision and face recognition technology, because it directly affects the accurate capture and analysis of facial features. The forehead area not only contains key information, but also plays a key role in applications such as facial expression recognition, emotion analysis, and identity authentication. With the widespread application of face recognition technology, especially in the fields of security monitoring, financial payment, and smart device unlocking, it is particularly important to accurately locate the forehead area. Correcting the forehead area can significantly improve the robustness and recognition accuracy of the algorithm, thereby ensuring the effectiveness and reliability of the system in complex environments.

[0004] Existing technologies face multiple challenges in correcting the position of the forehead area of ​​the face, mainly including insufficient adaptability to different face angles, expression changes and lighting conditions. In practical applications, due to the diversity and complexity of faces, traditional detection algorithms often have difficulty accurately identifying and correcting the forehead area, resulting in deviations in the position of feature points. In addition, existing methods usually rely on static models and lack dynamic adjustment capabilities, so they are prone to mispositioning or missed detection when processing real-time video streams or multi-angle inputs.

[0005] Therefore, how to accurately correct the forehead position and reduce the tracking error in the forehead area is an urgent problem to be solved. Summary of the invention

[0006] In view of this, the present invention provides a forehead position correction method, device, equipment and medium combined with depth calculation to solve the problem that the forehead position cannot be accurately corrected in the prior art and the tracking error of the forehead area is large.

[0007] The technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a forehead position correction method combined with depth calculation, the method comprising:

[0009] Perform face detection and face tracking on real-time images containing faces in nursing scenarios to determine initial forehead position information and face key point position information;

[0010] Determine the reference point position information through the nose key point position information and the mouth key point position information in the face key point position information;

[0011] Input the real-time image into the depth calculation model, and determine the reference point depth information according to the nose key point depth information and the mouth key point depth information;

[0012] Determine the displacement and depth change of the reference point in adjacent frame images according to the first reference point position information and the first reference point depth information in the first image, combined with the second reference point position information and the second reference point depth information in the second image, wherein the first image and the second image are adjacent real-time images;

[0013] The face state is determined based on the displacement and the depth change. When it is determined that the face is deformed, the initial forehead position information is corrected to determine the target forehead position information.

[0014] Preferably, the performing face detection and face tracking on the real-time image containing the face in the nursing scene to determine the initial forehead position information and the face key point position information includes:

[0015] Input the real-time image into a pre-trained face detection model, and output the face frame position information and the face key point position information;

[0016] Using a preset target tracking algorithm, the face frame position information in each frame of real-time image is tracked to output a face frame position sequence;

[0017] The face frame position sequence is calculated to determine the initial forehead position information in each frame of real-time image.

[0018] Preferably, the facial state is determined based on the displacement and the depth change, and when it is determined that the face is deformed, the initial forehead position information is corrected, and the target forehead position information is determined, including:

[0019] Obtaining a preset displacement change threshold and depth change threshold;

[0020] When the displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, it is determined that the face is deformed, the initial forehead position information is corrected, and the target forehead position information is determined.

[0021] Preferably, when the displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, it is determined that the face is deformed, the initial forehead position information is corrected, and the target forehead position information is determined, including:

[0022] Determining a depth change ratio according to the first reference point depth information and the second reference point depth information;

[0023] If it is determined that the face is deformed, the face deformation is classified to determine the deformation category;

[0024] According to the deformation type, combined with the depth change ratio and the displacement, the initial forehead position information is corrected to determine the target forehead position information.

[0025] Preferably, the correcting the initial forehead position information based on the deformation type and in combination with the depth change ratio and the displacement to determine the target forehead position information includes:

[0026] According to the deformation category, if the displacement is less than the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated by using the depth change ratio and the initial forehead position information;

[0027] If the displacement is greater than or equal to the displacement change threshold and the depth change is less than the depth change threshold, the target forehead position information is calculated by using the displacement and the initial forehead position information;

[0028] If the displacement is greater than or equal to the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated through the displacement, the depth change ratio and the initial forehead position information.

[0029] Preferably, after obtaining the preset displacement change threshold and depth change threshold, the method further includes:

[0030] If the displacement is smaller than the displacement change threshold and the depth change is smaller than the depth change threshold, the determination result is that the face is not deformed.

[0031] Preferably, after the step of determining that the face is not deformed if the displacement is less than the displacement change threshold and the depth change is less than the depth change threshold, the method further comprises:

[0032] If it is determined that the human face has not been deformed, the initial forehead position information is directly used as the target forehead position information.

[0033] In a second aspect, the present invention provides a forehead position correction device combined with depth calculation, the device comprising:

[0034] The face detection and tracking module is used to detect and track the real-time images containing faces in the nursing scene, and determine the initial forehead position information and the position information of the key points of the face;

[0035] A reference point position information determination module is used to determine the reference point position information through the nose key point position information and the mouth key point position information in the face key point position information;

[0036] A depth calculation module, used for inputting the real-time image into the depth calculation model, and determining reference point depth information according to the nose key point depth information and the mouth key point depth information;

[0037] A displacement and depth change determination module, used to determine the displacement and depth change of the reference point in adjacent frame images based on the first reference point position information and the first reference point depth information in the first image, combined with the second reference point position information and the second reference point depth information in the second image, wherein the first image and the second image are adjacent real-time images;

[0038] The forehead position correction module is used to determine the face state according to the displacement and the depth change. When it is determined that the face is deformed, the initial forehead position information is corrected to determine the target forehead position information.

[0039] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.

[0040] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.

[0041] In summary, the beneficial effects of the present invention are as follows:

[0042] The forehead position correction method, device, equipment and medium combined with depth calculation provided by the present invention include: performing face detection and face tracking on a real-time image containing a face in a nursing scene to determine initial forehead position information and face key point position information; determining reference point position information through nose key point position information and mouth key point position information in the face key point position information; inputting the real-time image into the depth calculation model, and determining reference point depth information according to the nose key point depth information and mouth key point depth information; determining the displacement and depth change of the reference point in adjacent frame images according to the first reference point position information and the first reference point depth information in the first image, combined with the second reference point position information and the second reference point depth information in the second image, wherein the first image and the second image are adjacent real-time images; judging the face state according to the displacement and the depth change, and when it is determined that the face is deformed, correcting the initial forehead position information to determine the target forehead position information. The present invention. First, the initial forehead position and facial key points are quickly determined through real-time image acquisition and efficient face detection and tracking technology. Second, the reference point position and depth information are accurately calculated by combining facial key point information and depth calculation models. This step takes into account the three-dimensional structure of the face and helps to more accurately adjust and estimate the actual position of the forehead to adapt to changes in different angles and expressions. Then, using this precise reference information, the initial forehead position is corrected to ensure more accurate positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0044] Figure 1 It is a schematic diagram of the overall working process of the forehead position correction method combined with depth calculation in Example 1 of the present invention;

[0045] Figure 2 This is a schematic diagram of the process of performing face detection and face tracking on a real-time image in Embodiment 1 of the present invention;

[0046] Figure 3 It is a schematic diagram of a process for determining reference point position information and reference point depth information in Embodiment 1 of the present invention;

[0047] Figure 4 This is a schematic diagram of a process for correcting initial forehead position information in Embodiment 1 of the present invention;

[0048] Figure 5This is a schematic diagram of a process for determining a face state in Embodiment 1 of the present invention;

[0049] Figure 6 This is a flow chart of correcting the initial forehead position information and determining the target forehead position information according to the determination result in Embodiment 1 of the present invention if it is determined that the face is deformed;

[0050] Figure 7 This is a schematic diagram of a process of correcting the initial forehead position information according to the deformation type, combined with the depth change ratio and the displacement in Embodiment 1 of the present invention;

[0051] Figure 8 is a structural block diagram of a forehead position correction device combined with depth calculation in Embodiment 2 of the present invention;

[0052] Fig. 9 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.

[0054] Example 1

[0055] See also Figure 1Embodiment 1 of the present invention discloses a forehead position correction method combined with depth calculation, the method comprising:

[0056] Obtain real-time images containing human faces in nursing scenarios;

[0057] Specifically, in daily care scenarios, a color video stream containing a human face is obtained, and a camera or other image acquisition device is used to record the facial expressions and dynamic changes of the subject to be tested in real time. By capturing color video, the subtle changes in skin color and optical properties are recorded in detail, which is crucial for subsequent facial recognition, emotion analysis, and extraction of physiological signals, such as heart rate measurement. In addition, the color video stream also provides enough details to support complex image processing and analysis techniques, so that faces can still be accurately identified and tracked in dynamic and changing environments, ensuring the quality and accuracy of care.

[0058] Performing face detection and face tracking on the real-time image to determine initial forehead position information and facial key point position information;

[0059] Specifically, in the care scenario, face detection technology is first used to identify faces in real-time images, and the position and size of the face are located by analyzing the real-time image data. Subsequently, face tracking technology is used to continuously monitor the detected face to ensure that the movement of the face can be tracked even in dynamic scenes. In addition, through advanced image processing algorithms, the positions of key points on the face, such as the eyes, nose, and mouth, can also be identified. The accurate identification of these key points is crucial to determining the position information of the forehead, because the forehead area is usually used for further physiological signal analysis, such as heart rate measurement.

[0060] In one embodiment, see Figure 2 The performing of face detection and face tracking on the real-time image to determine the initial forehead position information and the face key point position information includes:

[0061] Input the real-time image into a pre-trained face detection model, and output the face frame position information and the face key point position information;

[0062] Specifically, the real-time image is first input into a pre-trained face detection model, such as MTCNN. MTCNN is an efficient face detection method designed for accurately identifying faces and their key feature points in images. MTCNN accurately detects the face position and size in sequence through three cascaded convolutional neural networks, outputs the face frame F(x, y, w, h), and identifies the structural key points of the face such as the position of the eyes, nose and mouth. This step not only provides the necessary initial data for subsequent tracking and analysis, but also ensures the accurate capture of facial features, thus laying the foundation for accurate physiological signal analysis.

[0063] Using a preset target tracking algorithm, the face frame position information in each frame of real-time image is tracked to output a face frame position sequence;

[0064] Specifically, a target tracking algorithm such as the KCF algorithm is used to continuously track the detected face. KCF uses Fourier transform to perform correlation calculations in the frequency domain, which can efficiently handle changes in the face position in the video frame and maintain continuous tracking of the target face. By continuously updating the tracking information, a time series of face positions is output, which is crucial for maintaining stable monitoring of faces in dynamic environments, especially when the target has slight movements or the video quality changes.

[0065] The face frame position sequence is calculated to determine the initial forehead position information in each frame of real-time image.

[0066] Specifically, the position of the forehead is determined according to the face frame position sequence obtained from the face tracking process. According to the preset formula, starting from the position F(x, y, w, h) of the face frame, the approximate position E(x, y, w, h) of the forehead is calculated. The calculation formula is as follows:

[0067] E(x)=F(x)+F(w)*0.5-(F(w)*0.25 / 2.0)

[0068] E(y)=F(y)+F(h)*0.18-(F(w)*0.15 / 2.0)

[0069] E(w)=F(w)*0.25

[0070] E(h)=F(h)*0.15

[0071] This calculation takes into account the geometric structure characteristics of the human face. For example, the forehead is usually located in the central area above the face. The specific position and size are determined by the size ratio of the face frame. In this way, the forehead position in each frame of the image can be accurately calculated and updated, thereby ensuring that the data collection area is accurate and consistent in the subsequent heart rate analysis, increasing the accuracy and reliability of heart rate measurement.

[0072] Determine reference point position information and reference point depth information based on the facial key point position information and a preset depth calculation model;

[0073] In one embodiment, see Figure 3 The method of determining the reference point position information and the reference point depth information based on the facial key point position information and the preset depth calculation model includes:

[0074] Obtain the nose key point position information and the mouth key point position information in the face key point position information;

[0075] Specifically, through the above MTCNN model, the key points of the face are accurately detected, including the position information of the nose and mouth, and the position information of the nose key point N(x,y) and the position information of the two key points of the mouth M1(x,y) and M2(x,y) are output. The position information of these key points is crucial for the subsequent calculation of the reference position.

[0076] Determine the reference point position information according to the nose key point position information and the mouth key point position information;

[0077] Specifically, according to the position information of the nose and mouth key points, their coordinate centers are calculated as the reference position Ref(x, y). The reference position is calculated by taking the average value of the nose and mouth coordinate positions. The calculation formula is as follows:

[0078] Ref(x)=(N(x)+M1(x)+M2(x)) / 3

[0079] Ref(y)=(N(y)+M1(y)+M2(y)) / 3

[0080] In this way, the reference point position information in the center of the face is obtained for subsequent depth calculation.

[0081] Input the real-time image into the depth calculation model, and output the depth information of the key points of the nose and the mouth;

[0082] Specifically, a real-time image containing a human face is fed into a pre-trained depth calculation model, such as the MobileDepth monocular depth field model. The model can predict the depth information of each key point in the image based on the input image data, including the depth value D_N of the nose key point and the depth values ​​D_M1 and D_M2 of the mouth key points.

[0083] The reference point depth information is determined based on the nose key point depth information and the mouth key point depth information.

[0084] Specifically, according to the depth information of the nose and mouth key points obtained from the depth calculation model, their average depth value is calculated as the depth information of the reference point. The depth value of the reference point can be obtained by taking the average of the depth values ​​of the nose and mouth key points, that is, D_ref = (D_N + D_M1 + D_M2) / 3. This average depth value will be used as a reference for subsequent forehead correction to ensure the accuracy and stability of the forehead position.

[0085] Correcting the initial forehead position information according to the reference point position information and the reference point depth information to determine the target forehead position information;

[0086] Specifically, the initial forehead position information is corrected according to the reference point position information and the reference point depth information to determine the target forehead position information. This correction process is based on the position and depth of the reference point. By adjusting the initial forehead position to ensure the stability and accuracy of the forehead position in dynamic scenes, the corrected target forehead position information will better adapt to the posture and position changes of the face, thereby improving the accuracy and reliability of subsequent heart rate analysis. This correction method takes into account the geometric structure and depth characteristics of the face, and can effectively optimize the forehead position under different conditions, providing more reliable data support for heart rate measurement.

[0087] In one embodiment, see Figure 4 The method of correcting the initial forehead position information based on the reference point position information and the reference point depth information to determine the target forehead position information includes:

[0088] Acquire a first image and a second image adjacent to each other in a real-time image;

[0089] Specifically, in real-time images, a first image and a second image that are continuously captured and considered to be adjacent are obtained. These images are usually captured continuously at a certain frame rate, such as 30 frames per second or 60 frames per second, to ensure sufficient temporal resolution to capture the movement and changes of the face and key points. The two adjacent images are continuous in time and may have slight changes in space, such as the movement of the person or the adjustment of the posture.

[0090] Determine the displacement and depth change of the reference point in adjacent frame images based on the first reference point position information and the first reference point depth information in the first image and in combination with the second reference point position information and the second reference point depth information in the second image;

[0091] Specifically, first, the position information and depth information of the first reference point are obtained from the first image, and then the position information and depth information of the second reference point are obtained from the second image. Next, by comparing the position and depth information of the reference points in the two frames of images, the displacements Δx and Δy of the reference points on the image plane and the change ΔD in the depth direction are calculated. The calculation formula is as follows:

[0092] Δx=Ref2(x)-Ref1(x)

[0093] Δy=Ref2(y)-Ref1(y)

[0094] ΔD=D2_ref-D1_ref

[0095] The displacement Δx and Δy of the reference point on the image plane and the change ΔD in the depth direction reflect the movement and position adjustment of the reference point in adjacent frame images, which is of great significance for the subsequent forehead position correction. Such analysis helps to track the changes of the face in time and space, thereby achieving more accurate posture estimation and depth analysis.

[0096] Determining the face state according to the displacement and the depth change, and determining a determination result;

[0097] Specifically, the face state is determined based on the displacement and depth change to determine the determination result. By analyzing the movement and posture changes of the face between consecutive frames, the state evaluation and decision-making are carried out. By analyzing the displacement and depth change, the movement state of the face is inferred, such as whether offset, rotation or scaling has occurred. At the same time, the depth change can also provide information about the changes in the distance of the face. Based on these data, the face state is divided into different categories, such as whether deformation has occurred, or further subdivided into more specific states as needed, such as smiling, closing eyes, turning the head, etc. This state determination can provide an important reference for subsequent processing. For example, in heart rate measurement, a suitable feature area adjustment strategy is selected according to the face state, thereby improving the accuracy and stability of the measurement.

[0098] In one embodiment, see Figure 5 The facial state is determined based on the displacement and the depth change, and the determination result includes:

[0099] Obtaining a preset displacement change threshold and depth change threshold;

[0100] Specifically, a pre-set displacement change threshold (Theta1) and depth change threshold (Theta2) are obtained. The displacement change threshold and the depth change threshold are usually based on the understanding of the application scenario and the analysis of experimental results. The displacement change threshold reflects the maximum allowable range of the position change of the face in the image, while the depth change threshold represents the maximum allowable range of the distance change of the face. Usually, the displacement change threshold and the depth change threshold need to be adjusted according to the specific application requirements and actual data to ensure that the face status can be accurately determined.

[0101] If the displacement is less than the displacement change threshold and the depth change is less than the depth change threshold, then the determination result is that the face is not deformed;

[0102] Specifically, when both the calculated displacement and depth change are less than the preset thresholds, i.e., Δx|+|Δy|<Theta1 and ΔD<Theta2, it is determined that the face has not deformed. At this time, it is considered that the position and shape of the face region remain stable, and no adjustment of the forehead region is required.

[0103] If the displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, the determination result is that the face has deformed.

[0104] Specifically, if the calculated displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, i.e., Δx|+|Δy|>=Theta1 and / or ΔD>=Theta2, it will be determined that the face has deformed. This indicates that there has been a significant change in the position or depth of the face region, and corresponding adjustments are required to ensure the accuracy of subsequent processing. In this case, the forehead region is corrected accordingly to adapt to the change of the face, so as to ensure that reliable results can be obtained in subsequent biometric signal measurements.

[0105] According to the determination result, if it is determined that the face has deformed, the initial forehead position information is corrected to determine the target forehead position information;

[0106] Specifically, according to the determination result, if it is determined that the face has deformed, the initial forehead position information needs to be corrected to determine the target forehead position information. This means applying a preset correction strategy to adjust the current forehead position to adapt to the new position and shape of the face. The correction strategy includes transformations such as translation, scaling, or rotation of the forehead region to ensure that the forehead region remains stable and aligned at the new face position. Through this correction, the accuracy of face tracking and pose estimation is improved, so as to ensure that reliable results can be obtained in subsequent biometric signal measurements under dynamic face conditions.

[0107] In one embodiment, please refer to Figure 6 , where according to the determination result, if it is determined that the face has deformed, the correction of the initial forehead position information to determine the target forehead position information includes:

[0108] Determine the depth change ratio based on the first reference point depth information and the second reference point depth information;

[0109] Specifically, use the depth information of the first and second reference points to calculate the depth change ratio, and its calculation formula is:

[0110] Ratio=D2_ref / D1_ref

[0111] The depth change ratio (Ratio) indicates the depth change degree of the second frame image relative to the depth change degree of the first frame image. By comparing the depth information of the reference point in the two frames of images, the depth change ratio is calculated, which will become an important parameter for subsequent forehead position correction.

[0112] If it is determined that the face is deformed, the face deformation is classified to determine the deformation category;

[0113] Specifically, if the system determines that the face is deformed, that is, the corresponding displacement and depth change threshold conditions are met, then the system needs to classify the deformation. This includes deformations in different directions, such as horizontal, vertical, or depth changes. According to the size and direction of the displacement and depth changes, the deformation is classified into different situations, such as only depth movement, only horizontal / vertical movement, or depth and translation at the same time.

[0114] According to the deformation type, combined with the depth change ratio and the displacement, the initial forehead position information is corrected to determine the target forehead position information.

[0115] Specifically, the initial forehead position information is corrected according to the deformation type, depth change ratio and displacement to determine the target forehead position information. The correction process makes different adjustments according to different deformation situations. For example, if only depth movement occurs, the size and position of the forehead will be adjusted using the depth change ratio; if horizontal or vertical movement occurs, the position of the forehead will be adjusted using the displacement. Through such a correction process, the deformation of the face is adapted and the forehead position is ensured to remain stable and accurate in continuous frame images, thereby improving the accuracy and reliability of subsequent biosignal measurements.

[0116] In one embodiment, see Figure 7 The initial forehead position information is corrected based on the deformation type, combined with the depth change ratio and the displacement, and the target forehead position information is determined, including:

[0117] According to the deformation category, if the displacement is less than the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated by using the depth change ratio and the initial forehead position information;

[0118] Specifically, when the displacement is less than the displacement change threshold and the depth change is greater than or equal to the depth change threshold, and the deformation category is determined to be a case where the displacement is small, the depth change ratio and the initial forehead position information are used to calculate the target forehead position information. The depth change ratio is used to determine the degree of change in the depth of the face, and then the new target forehead position is calculated based on the initial forehead position information and the influence of the depth change. The calculation formula is:

[0119] E2(x)=E1(x)+E1(w) / 2-E1(w)*Ratio / 2.0;

[0120] E2(y)=E1(y)+E1(y) / 2-E1(h)*Ratio / 2.0

[0121] E2(w)=E1(w)*Ratio

[0122] E2(h)=E1(h)*Ratio

[0123] This correction process ensures that the position of the forehead area in the depth direction is consistent with the changes in the face, thereby improving the stability and accuracy of the forehead position.

[0124] If the displacement is greater than or equal to the displacement change threshold and the depth change is less than the depth change threshold, the target forehead position information is calculated by using the displacement and the initial forehead position information;

[0125] Specifically, when the displacement is greater than or equal to the displacement change threshold and the depth change is less than the depth change threshold, the deformation category is horizontal or vertical movement, but the depth change is small, the displacement and the initial forehead position information are used to calculate the target forehead position information. The displacement is used to determine the degree of horizontal or vertical movement of the face, and then the new target forehead position is calculated based on the initial forehead position information and the influence of the displacement. The calculation formula is:

[0126] E2(x)=E1(x)+Δx

[0127] E2(y)=E1(y)+Δy

[0128] E2(w)=E1(w)

[0129] E2(h)=E1(h)

[0130] This correction process ensures that the forehead area is consistent with the changes in the human face in the horizontal or vertical direction, thereby improving the stability and accuracy of the forehead position.

[0131] If the displacement is greater than or equal to the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated through the displacement, the depth change ratio and the initial forehead position information.

[0132] Specifically, when the displacement is greater than or equal to the displacement change threshold and the depth change is greater than or equal to the depth change threshold, that is, when the deformation category is depth movement and the displacement is large, the displacement, depth change ratio and initial forehead position information are considered simultaneously to calculate the target forehead position information. The displacement and depth change ratio are used to comprehensively consider the degree of change of the face in the horizontal, vertical and depth directions, and then the new target forehead position is calculated based on the initial forehead position information. The calculation formula is:

[0133] E2(x)=E1(x)+E1(w) / 2-E1(w)*Ratio / 2.0+Δx;

[0134] E2(y)=E1(y)+E1(y) / 2-E1(h)*Ratio / 2.0+Δy;

[0135] E2(w)=E1(w)*Ratio;

[0136] E2(h)=E1(h)*Ratio;

[0137] This correction process that takes different factors into consideration can more accurately adapt to the deformation of the face, thereby improving the stability and accuracy of the forehead position.

[0138] Specifically, the advantage of the above-mentioned forehead area correction scheme over forehead area tracking by filtering is that it can more accurately adapt to the deformation and posture changes of the face. By comprehensively considering the displacement, depth change ratio and initial forehead position information, the changes in the face area can be captured more comprehensively, so that the target forehead position can be calculated more accurately. In comparison, for example, the filtering method combining Mean-Shift and Kalman is more susceptible to noise interference, and the forehead area cannot be effectively tracked when the face deformation is large or the posture changes are large. Therefore, this correction scheme can provide more stable and accurate forehead position information, thereby obtaining more reliable results in applications such as heart rate measurement.

[0139] If it is determined that the human face has not been deformed, the initial forehead position information is directly used as the target forehead position information.

[0140] Based on the target forehead position information, a heart rate analysis is performed on the forehead area to determine the heart rate of the subject to be measured.

[0141] Specifically, according to the target forehead position information, the forehead area is accurately located and used as the area of ​​interest for heart rate analysis. By using non-contact optical technology, such as photoplethysmography (PPG), the tiny blood flow in the forehead area is captured and the heartbeat signal is extracted from it. By analyzing the frequency and change pattern of these heartbeat signals, the heart rate of the subject to be measured is accurately calculated. This method does not require direct contact with the skin, is non-invasive, convenient and fast, and is suitable for real-time monitoring of the heart rate of the subject to be measured in daily care scenarios.

[0142] Example 2

[0143] See also Figure 8 Embodiment 2 of the present invention further provides a forehead position correction device combined with depth calculation, the device comprising:

[0144] An image acquisition module is used to acquire real-time images containing human faces in nursing scenarios;

[0145] A face detection and tracking module, used to perform face detection and face tracking on the real-time image, and determine initial forehead position information and face key point position information;

[0146] A reference point information acquisition module, used to determine reference point position information and reference point depth information based on the facial key point position information and in combination with a preset depth calculation model;

[0147] A position correction module, used to correct the initial forehead position information according to the reference point position information and the reference point depth information, and determine the target forehead position information;

[0148] The heart rate analysis module is used to perform heart rate analysis on the forehead area according to the target forehead position information to determine the heart rate of the subject to be measured.

[0149] Specifically, a forehead position correction device combined with depth calculation provided by an embodiment of the present invention is adopted, and the device includes: an image acquisition module for acquiring a real-time image containing a face in a nursing scene; a face detection and tracking module for performing face detection and face tracking on the real-time image to determine the initial forehead position information and the face key point position information; a reference point information acquisition module for determining the reference point position information and the reference point depth information based on the face key point position information and the preset depth calculation model; a position correction module for correcting the initial forehead position information based on the reference point position information and the reference point depth information to determine the target forehead position information; a heart rate analysis module for performing heart rate analysis on the forehead area based on the target forehead position information to determine the heart rate of the object to be measured. This device greatly improves the measurement accuracy and reduces the positioning error of the forehead area by stabilizing the heart rate measurement. First, the initial forehead position and face key points are quickly determined through real-time image acquisition and efficient face detection and tracking technology, which provides a basic spatial positioning for heart rate measurement. Secondly, by combining the facial key point information and the depth calculation model, the reference point position and depth information are accurately calculated. This step takes into account the three-dimensional structure of the face, which helps to more accurately adjust and estimate the actual position of the forehead to adapt to changes in different angles and expressions. Then, using this precise reference information, the initial forehead position is corrected to ensure more accurate positioning, which is crucial for the acquisition of PPG signals because PPG signals are very sensitive to the accuracy of the detection position. Finally, by accurately correcting the forehead position and performing heart rate analysis, the subtle optical changes caused by blood flow can be more effectively captured, thereby improving the accuracy of heart rate measurement.

[0150] Example 3

[0151] In addition, combined Figure 1 The forehead position correction method combined with depth calculation of the embodiment 1 of the present invention described above can be implemented by an electronic device. Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 3 of the present invention is shown.

[0152] The electronic device may include a processor and a memory storing computer program instructions.

[0153] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0154] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0155] The processor implements any one of the forehead position correction methods combined with depth calculation in the above embodiments by reading and executing computer program instructions stored in the memory.

[0156] In one example, the electronic device may further include a communication interface and a bus. Fig. 9 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0157] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0158] The bus includes hardware, software or both, and the parts of the device are coupled to each other. For example, but not limitation, the bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, the bus may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0159] Example 4

[0160] In addition, in combination with the forehead position correction method combined with depth calculation in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any forehead position correction method combined with depth calculation in the above embodiments is implemented.

[0161] In summary, the embodiments of the present invention provide a forehead position correction method, device, equipment and medium combined with depth calculation.

[0162] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0163] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0164] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0165] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0166] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A forehead position correction method combined with depth calculation, characterized in that: The method comprises: Perform face detection and face tracking on real-time images containing faces in nursing scenarios to determine initial forehead position information and face key point position information; Determine the reference point position information through the nose key point position information and the mouth key point position information in the face key point position information; Input the real-time image into the depth calculation model, and determine the reference point depth information according to the nose key point depth information and the mouth key point depth information; Determine the displacement and depth change of the reference point in adjacent frame images according to the first reference point position information and the first reference point depth information in the first image, combined with the second reference point position information and the second reference point depth information in the second image, wherein the first image and the second image are adjacent real-time images; The face state is determined based on the displacement and the depth change. When it is determined that the face is deformed, the initial forehead position information is corrected to determine the target forehead position information.

2. The forehead position correction method combined with depth calculation according to claim 1, characterized in that: The performing of face detection and face tracking on the real-time image containing the face in the nursing scene to determine the initial forehead position information and the face key point position information includes: Input the real-time image into a pre-trained face detection model, and output the face frame position information and the face key point position information; Using a preset target tracking algorithm, the face frame position information in each frame of real-time image is tracked to output a face frame position sequence; The face frame position sequence is calculated to determine the initial forehead position information in each frame of real-time image.

3. The forehead position correction method combined with depth calculation according to claim 1, characterized in that: The determining of the face state according to the displacement and the depth change, and when it is determined that the face is deformed, correcting the initial forehead position information, and determining the target forehead position information includes: Obtaining a preset displacement change threshold and depth change threshold; When the displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, it is determined that the face is deformed, the initial forehead position information is corrected, and the target forehead position information is determined.

4. The forehead position correction method combined with depth calculation according to claim 3, characterized in that: When the displacement is greater than or equal to the displacement change threshold and / or the depth change is greater than or equal to the depth change threshold, determining that the face is deformed, correcting the initial forehead position information, and determining the target forehead position information includes: Determining a depth change ratio according to the first reference point depth information and the second reference point depth information; If it is determined that the face is deformed, the face deformation is classified to determine the deformation category; According to the deformation type, combined with the depth change ratio and the displacement, the initial forehead position information is corrected to determine the target forehead position information.

5. The forehead position correction method combined with depth calculation according to claim 4, characterized in that: The correcting the initial forehead position information according to the deformation type and in combination with the depth change ratio and the displacement to determine the target forehead position information includes: According to the deformation category, if the displacement is less than the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated by using the depth change ratio and the initial forehead position information; If the displacement is greater than or equal to the displacement change threshold and the depth change is less than the depth change threshold, the target forehead position information is calculated by using the displacement and the initial forehead position information; If the displacement is greater than or equal to the displacement change threshold and the depth change is greater than or equal to the depth change threshold, the target forehead position information is calculated through the displacement, the depth change ratio and the initial forehead position information.

6. The forehead position correction method combined with depth calculation according to claim 3, characterized in that: After obtaining the preset displacement change threshold and depth change threshold, the method further includes: If the displacement is smaller than the displacement change threshold and the depth change is smaller than the depth change threshold, the determination result is that the face is not deformed.

7. The forehead position correction method combined with depth calculation according to claim 6, characterized in that: After the step of determining that the face is not deformed if the displacement is less than the displacement change threshold and the depth change is less than the depth change threshold, the step further includes: If it is determined that the human face has not been deformed, the initial forehead position information is directly used as the target forehead position information.

8. A forehead position correction device combined with depth calculation, characterized in that: The device comprises: The face detection and tracking module is used to detect and track the real-time images containing faces in the nursing scene, and determine the initial forehead position information and the position information of the key points of the face; A reference point position information determination module is used to determine the reference point position information through the nose key point position information and the mouth key point position information in the face key point position information; A depth calculation module, used for inputting the real-time image into the depth calculation model, and determining reference point depth information according to the nose key point depth information and the mouth key point depth information; A displacement and depth change determination module, used to determine the displacement and depth change of the reference point in adjacent frame images based on the first reference point position information and the first reference point depth information in the first image, combined with the second reference point position information and the second reference point depth information in the second image, wherein the first image and the second image are adjacent real-time images; The forehead position correction module is used to determine the face state according to the displacement and the depth change. When it is determined that the face is deformed, the initial forehead position information is corrected to determine the target forehead position information.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images