A multi-target key head pose extraction method based on visual target guidance

Abstract

The application provides a multi-target key head posture extraction method based on a visual target guide, which comprises the following steps: performing face detection and inter-frame correlation on a target person to obtain a target trajectory sequence; performing key point detection and template matching on the target trajectory sequence to obtain a standard face template; performing affine alignment on the target trajectory sequence through the standard face template to obtain an aligned image; solving the standard face template by minimizing a projection error function to obtain three-dimensional posture parameters; performing difference on a three-dimensional position sequence of the aligned image to obtain a motion direction; calculating head feature parameters according to the three-dimensional posture parameters and constructing a weighted average model to screen key targets of the aligned image to obtain a key head posture result. Through multi-target face detection, geometric posture estimation and visual target prior modeling, combined with time sequence correlation analysis, the method significantly improves the recognition accuracy and posture estimation stability of key targets in a multi-person scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction intelligent perception technology, and in particular to a method for extracting key head poses of multiple targets based on visual target guidance. Background Technology

[0002] In multi-target video monitoring and gaze training, the system needs to accurately identify key individuals who are truly looking at the target within the same video frame in order to achieve effective gaze guidance and behavior analysis. However, traditional head pose estimation and gaze detection methods still face the following key technical challenges in multi-target scenarios: 1. Difficulty in attitude determination under multi-target interference Existing methods typically perform pose estimation for each face individually, without fully considering the presence of multiple individuals in the scene. Since the head directions of different individuals may be similar or briefly overlap, traditional methods rely solely on image features or angle thresholds to determine whether a target is gazing at the target, which can easily lead to misidentification or duplicate judgments, causing the system to be unable to accurately distinguish between "key gazers" and "non-gazers".

[0003] 2. Insufficient utilization of prior visual targets In actual fixation training, the spatial position and trajectory of the target are known prior information. Existing methods neglect the spatial consistency relationship between the target's geometric position and facial orientation. This lack of correlation modeling results in pose estimation results that fail to reflect the true spatial intent of the fixated target, and also prevents dynamic adjustment of pose judgment criteria as the target moves.

[0004] 3. Lack of effective relevance calculation and key target screening mechanisms In multi-target scenarios, accurately identifying the target that best matches the gaze target's movement among all detected faces is a core issue affecting system performance. Existing solutions mostly rely on static angle thresholds or manually set weights, which cannot establish a temporally continuous correlation evaluation mechanism under complex motion conditions. In particular, when multiple targets alternately gaze at the gaze target, the system often experiences temporal drift or misselection.

[0005] 4. Insufficient time consistency and noise suppression Traditional pose estimation relies solely on single-frame or short-term features, failing to model the gaze consistency of the target over time. In real-world video streams, variations in lighting, facial occlusion, and detection errors can all cause instantaneous angle fluctuations, affecting the stability of pose determination. The lack of a time-weighted smoothing-based correlation scoring mechanism further hinders the system's ability to consistently output key targets in dynamic environments. Summary of the Invention

[0006] The purpose of this invention is to provide a method for extracting key head poses of multiple targets based on optotype guidance. By combining multi-target face detection, geometric pose estimation and optotype prior modeling with temporal correlation analysis, the method can achieve automatic identification and head orientation estimation of key individuals in multi-target scenarios.

[0007] To achieve the above objectives, the present invention provides the following solution: A method for extracting key head poses of multiple targets based on visual target guidance includes the following steps: Face detection and inter-frame correlation are performed on the target person to obtain the target trajectory sequence; Key point detection and template matching are performed on the target trajectory sequence to obtain a standard face template; The target trajectory sequence is affinely aligned using a standard face template to obtain an aligned image; The three-dimensional pose parameters are obtained by solving the standard face template by minimizing the projection error function; The direction of motion is obtained by differentiating the three-dimensional position sequence of the aligned image; The head feature parameters are calculated based on the three-dimensional pose parameters, and a weighted average model is constructed based on the head feature parameters. The key head pose results are obtained by filtering key targets in the aligned image using a weighted average model.

[0008] Optionally, face detection and inter-frame correlation are performed on the target person to obtain a target trajectory sequence, including: The video stream of the target person is captured by a camera, and the pixel motion field of the video stream is calculated. Perform face detection on the video stream to obtain a set of detection boxes; The target trajectory sequence is obtained by performing inter-frame correlation on the detection box set using the SORT framework.

[0009] Optionally, keypoint detection and template matching are performed on the target trajectory sequence to obtain a standard face template, including: Feature points are extracted from the target trajectory sequence using a feature point detection algorithm to obtain a two-dimensional feature point set; the feature points include: corners of the eyes, tip of the nose, corners of the mouth, and chin; Standard three-dimensional coordinates of feature points are selected from the average face template using a set of two-dimensional feature points. A standard face template is constructed based on standard three-dimensional coordinates.

[0010] Optionally, the target trajectory sequence is affine aligned using a standard face template to obtain an aligned image, including: The horizontal reference center of the face is calculated based on the set of two-dimensional feature points. The rotation angle is determined based on the set of two-dimensional feature points, and the first rotation matrix is ​​constructed based on the rotation angle. The rotation matrix is ​​solved using the least squares method to obtain the translation term, which is then aligned with the center of the standard face template to obtain the aligned image.

[0011] Optionally, the three-dimensional attitude parameters include: a second rotation matrix and a translation vector; the formula for solving the three-dimensional attitude parameters is: ;in, For perspective projection functions, For feature point weights, It is a set of two-dimensional feature points. This is the second rotation matrix. As a standard face template, For three-dimensional pose, It is a translation vector.

[0012] Optionally, the formula for calculating the direction of motion is: ;in, This is the current three-dimensional position sequence. Time interval from the current moment The three-dimensional position sequence.

[0013] Optionally, parameters are calculated based on the three-dimensional pose parameters to obtain head feature parameters, and a weighted average model is constructed based on the head feature parameters, including: The center point is calculated based on the three-dimensional pose parameters to obtain the facial features; the facial features include: the center point of the face and the direction vector of the face relative to the target; The angle between the head direction and the target direction is calculated based on the three-dimensional posture parameters; A weighted average model is constructed based on the pixel motion field and the angle between the head direction and the target direction.

[0014] Alternatively, the expression for the weighted average model is: ;in, The motion intensity of pixels in adjacent frames. For pixel sports fields, For instantaneous correlation score, For instantaneous time index, For time window, As the first adjustment factor, The correlation score is weighted over time.

[0015] Optionally, key targets are filtered from the aligned image using a weighted average model to obtain key head pose results, including: The comprehensive correlation index is calculated based on the direction of motion; the formula for calculating the comprehensive correlation index is: ;in, The rate of change of the target velocity in the direction of motion. The second adjustment factor; The three-dimensional pose parameter with the highest score among the comprehensive relevant indicators is selected as the key head pose result.

[0016] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides a multi-target key head pose extraction method based on target guidance. This method includes: performing face detection and inter-frame correlation on the target person to obtain a target trajectory sequence; performing key point detection and template matching on the target trajectory sequence to obtain a standard face template; performing affine alignment on the target trajectory sequence using the standard face template to obtain an aligned image; solving the standard face template by minimizing the projection error function to obtain three-dimensional pose parameters; performing difference on the three-dimensional position sequence of the aligned image to obtain the motion direction; calculating parameters based on the three-dimensional pose parameters to obtain head feature parameters, and constructing a weighted average model based on the head feature parameters; and filtering key targets in the aligned image based on the weighted average model to obtain key head pose results. This method, through multi-target face detection, geometric pose estimation, and target prior modeling, combined with temporal correlation analysis, achieves automatic identification and head direction estimation of key individuals in multi-target scenarios, significantly improving the recognition accuracy and pose estimation stability of key targets in multi-person scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the multi-target key head pose extraction method according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] like Figure 1 As shown, this embodiment of the invention provides a method for extracting key head poses of multiple targets based on lookahead guidance, including the following steps: Step 100: Perform face detection and inter-frame correlation on the target person to obtain the target trajectory sequence; Step 200: Perform key point detection and template matching on the target trajectory sequence to obtain a standard face template; Step 300: Perform affine alignment on the target trajectory sequence using a standard face template to obtain an aligned image; Step 400: Solve the standard face template by minimizing the projection error function to obtain the three-dimensional pose parameters; Step 500: Difference the 3D position sequence of the aligned image to obtain the direction of motion; Step 600: Calculate the parameters based on the three-dimensional pose parameters to obtain the head feature parameters, and construct a weighted average model based on the head feature parameters; Step 700: Select key targets from the aligned image using a weighted average model to obtain key head pose results.

[0022] Specifically, face detection and inter-frame correlation are performed on the target person to obtain the target trajectory sequence, including: The video stream of the target person is captured by a camera, and the pixel motion field of the video stream is calculated. Perform face detection on the video stream to obtain a set of detection boxes; The target trajectory sequence is obtained by performing inter-frame correlation on the detection box set using the SORT framework.

[0023] In the specific implementation process, step 100 uses a single camera to capture a video stream containing multiple facial targets. The system establishes a unified timestamp (t) for each frame of the video stream and performs distortion correction and grayscale normalization during video acquisition to ensure consistent image brightness and scale. Simultaneously, it calculates the pixel motion field between adjacent frames using optical flow. Then, the YOLOv8 face detection algorithm, fine-tuned with facial data, is used to extract all face regions from each frame of the video stream, forming a set of detection boxes. To improve detection robustness, a confidence threshold is also set in some embodiments. Only confidence level is retained. The goal is to reduce false detection interference. Then, the SORT framework, combining Kalman filtering and the Hungarian algorithm, is used to perform inter-frame association of the detection box set, assigning a unique identifier to each detection box. Thus, the target trajectory sequence is constructed. This ensures the consistency of identity for the same face across different frames, with trajectory continuity achieved through Euclidean distance.

[0024] Specifically, key point detection and template matching are performed on the target trajectory sequence to obtain a standard face template, including: Feature points are extracted from the target trajectory sequence using a feature point detection algorithm to obtain a two-dimensional feature point set; Standard three-dimensional coordinates of feature points are selected from the average face template using a set of two-dimensional feature points. A standard face template is constructed based on standard three-dimensional coordinates.

[0025] In the specific implementation process, step 200 uses the traditional feature point detection algorithm Dlib to extract feature points from the target trajectory sequence, including structurally stable regions such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the chin, thereby forming a two-dimensional feature point set for each face. , This provides a reliable constraint on the number of feature points for 3D pose estimation. Then, using an average face template based on a 3D Morphable Model (3DMM), standard 3D coordinates corresponding to feature points are selected from a public dataset to construct a standard face 3D template point set (i.e., a standard face template). The point set and the two-dimensional facial feature points are used to establish a two-dimensional-three-dimensional matching relationship according to the numbering. The average facial template adopts a facial geometric model with a fixed ratio.

[0026] Specifically, the target trajectory sequence is affine-aligned using a standard face template to obtain an aligned image, including: The horizontal reference center of the face is calculated based on the set of two-dimensional feature points. The rotation angle is determined based on the set of two-dimensional feature points, and the first rotation matrix is ​​constructed based on the rotation angle. The rotation matrix is ​​solved using the least squares method to obtain the translation term, which is then aligned with the center of the standard face template to obtain the aligned image.

[0027] In the specific implementation process, step 300 addresses each objective. Affine alignment is performed based on the line connecting the eyes and the position of the nose tip to rotate the face to a uniform pose plane. First, from a set of two-dimensional feature points... Extracting the center of the left eye Right eye center and the tip of the nose Calculate the midpoint of the two eyes. Use this as a horizontal reference center for the face. Then determine the rotation angle based on the direction of the line connecting the two eyes. , The two-dimensional coordinates of the center of the right eye. Let the coordinates of the left eye center be two-dimensional, and construct the rotation matrix. The expression is: ; Subsequently, the corresponding feature point positions in the standard face template are defined as follows: Solving the affine matrix using the least squares method The expression is: ; in By rotation matrix The translation term is derived. This is used to align the tip of the nose to the center of a standard face template, resulting in an aligned image. Finally, the aligned image is scale-normalized to standardize the interocular distance to a fixed length. (in some embodiments) ), by scaling factor This yields an affine matrix of uniform scale. It should be noted that after step 300, faces of different individuals and poses are mapped to a standard reference plane with a uniform scale and orientation, providing stable input for subsequent 3D pose estimation.

[0028] Specifically, the three-dimensional pose parameters in step 400 include: rotation matrix. Translation vector All of these are obtained by minimizing the projection error function, which is expressed as: ; in, Represents three-dimensional pose. Represents the translation vector. For perspective projection functions, As the feature point weights, in some embodiments, the weight of structurally stable regions (nose tip) is 1.0, the weight of moderately stable regions such as the corner of the eye is 0.7, and the weight of regions easily affected by facial expressions such as the eyebrows is 0.4.

[0029] Specifically, step 500 involves aligning the three-dimensional position sequence of the image. By performing a difference operation, the direction of motion is obtained, expressed as: ; in, This is the current three-dimensional position sequence. Time interval from the current moment The three-dimensional position sequence. The vector of the motion direction describes the spatial motion trend of the target at the current moment, providing a dynamic geometric prior for subsequent correlation calculations.

[0030] Specifically, parameters are calculated based on the three-dimensional pose parameters to obtain head feature parameters, and a weighted average model is constructed based on the head feature parameters, including: The center point is calculated based on the three-dimensional pose parameters to obtain the facial features; the facial features include: the center point of the face and the direction vector of the face relative to the target; The angle between the head direction and the target direction is calculated based on the three-dimensional posture parameters; A weighted average model is constructed based on the pixel motion field and the angle between the head direction and the target direction.

[0031] In the specific implementation process, the face center point in step 600 The expression is: ; in, The center point of the template. The direction vector of the face relative to the viewpoint. The expression is: ; The orientation vector of a face relative to the viewpoint represents the spatial orientation relationship of each target relative to the viewpoint.

[0032] Then, based on the rotation matrix... Extract the forward unit vector of the head Then calculate the angle between the head direction and the target direction. , This represents the direction vector of the face relative to the target. The smaller the angle, the more likely the target is to be looking at the target.

[0033] Next, based on angular consistency, the instantaneous correlation score is defined as follows: To suppress instantaneous fluctuations caused by noise, within a time window ( Weighted smoothing is performed within the range. A temporal smoothing mechanism based on optical flow weights is introduced, and a weighted average model of pose over time is constructed by combining the pixel motion field. The expression is: ; in, Indicates an instantaneous time index. For time window, To adjust the factor, The correlation score is weighted over time. The motion intensity of pixels in adjacent frames is used to adaptively adjust the temporal smoothing weights. For pixel sports fields, This represents the instantaneous correlation score. When the exercise is more intense, A larger value increases the weight of recent frames, thereby suppressing detection jumps and maintaining attitude continuity. In some embodiments, the optical flow weight... In scenarios requiring higher responsiveness The value of can be increased, while it can be appropriately decreased in scenarios where stronger smoothness is required.

[0034] Specifically, key targets are screened from the aligned image using a weighted average model to obtain key head pose results, including: The comprehensive relevant indicators are calculated based on the direction of motion; The three-dimensional pose parameter with the highest score among the comprehensive relevant indicators is selected as the key head pose result.

[0035] In the specific implementation process, step 700 combines the instantaneous correlation score with the change in target velocity. The comprehensive relevant index is calculated using the following formula: ; in, The rate of change of the target velocity in the direction of motion. This is an adjustment factor. The target with the highest score is selected. Output the three-dimensional attitude parameters corresponding to the target. As a key head pose result, it ensures that the key person who best matches the target behavior characteristics can be identified in real time in a multi-target environment.

[0036] The beneficial effects of this invention are as follows: 1) By calculating the spatial geometric relationship between the head center point of multiple targets and the position of the target, and generating the fitting score of each target, the automatic recognition of key targets is realized. It not only considers the features of a single frame image, but also uses the target information to constrain the head posture, which significantly improves the recognition accuracy and posture estimation stability of key targets in multi-person scenes. 2) By establishing spatial position and pose correlations of multiple targets in video frame sequences and calculating the correlation changes between the target head and the visual target, inter-frame information fusion is realized, which effectively alleviates the impact of occlusion, rapid head movement and illumination changes on pose estimation and improves the robustness and robustness of detection in complex dynamic environments. 3) This invention relies entirely on geometric relationships and image feature calculations, and can achieve key target recognition and pose optimization without collecting and labeling a large amount of training data. This not only reduces data collection costs, but also improves interpretability and system deployment convenience, making it suitable for various application scenarios such as video surveillance, behavior analysis and human-computer interaction. 4) This invention can complete the extraction of key head poses of multiple targets and the calculation of visual object correlation with only a single RGB camera. Compared with traditional methods that rely on multiple cameras or complex deep learning networks, it significantly reduces the hardware and computing power costs of model training and inference, and is easy to deploy and run in real time in ordinary indoor environments.

[0037] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0038] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for extracting key head poses of multiple targets based on lookahead guidance, characterized in that, Includes the following steps: Face detection and inter-frame correlation are performed on the target person to obtain the target trajectory sequence; Key point detection and template matching are performed on the target trajectory sequence to obtain a standard face template; The target trajectory sequence is affinely aligned using the standard face template to obtain an aligned image; The three-dimensional pose parameters are obtained by solving the standard face template by minimizing the projection error function; The direction of motion is obtained by differentiating the three-dimensional position sequence of the aligned image; The head feature parameters are obtained by calculating the parameters based on the three-dimensional pose parameters, and the weighted average model is constructed based on the head feature parameters. The key target is filtered out from the aligned image based on the weighted average model to obtain the key head pose results.

2. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 1, characterized in that, Face detection and inter-frame correlation are performed on the target person to obtain the target trajectory sequence, including: The video stream of the target person is captured by a camera and the pixel motion field of the video stream is calculated. Face detection is performed on the video stream to obtain a set of detection boxes; The target trajectory sequence is obtained by performing inter-frame correlation on the detection box set using the SORT framework.

3. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 1, characterized in that, Key point detection and template matching are performed on the target trajectory sequence to obtain a standard face template, including: Feature points are extracted from the target trajectory sequence using a feature point detection algorithm to obtain a two-dimensional feature point set; the feature points include: corners of the eyes, tip of the nose, corners of the mouth, and chin; The standard three-dimensional coordinates of the feature points are selected from the average face template using the set of two-dimensional feature points. The standard face template is constructed based on the standard three-dimensional coordinates.

4. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 3, characterized in that, The target trajectory sequence is affinely aligned using the standard face template to obtain an aligned image, including: The horizontal reference center of the face is calculated based on the set of two-dimensional feature points. The rotation angle is determined based on the set of two-dimensional feature points, and a first rotation matrix is ​​constructed based on the rotation angle; The rotation matrix is ​​solved using the least squares method to obtain the translation term, which is then aligned with the center of the standard face template to obtain the aligned image.

5. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 1, characterized in that, The three-dimensional attitude parameters include: a second rotation matrix and a translation vector; the formula for solving the three-dimensional attitude parameters is: ;in, For perspective projection functions, For feature point weights, It is a set of two-dimensional feature points. This is the second rotation matrix. As a standard face template, For three-dimensional pose, It is a translation vector.

6. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 1, characterized in that, The formula for calculating the direction of motion is: ;in, This is the current three-dimensional position sequence. Time interval from the current moment The three-dimensional position sequence.

7. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 2, characterized in that, Based on the three-dimensional pose parameters, parameter calculations are performed to obtain head feature parameters, and the weighted average model is constructed based on the head feature parameters, including: The center point is calculated based on the three-dimensional pose parameters to obtain facial features; the facial features include: the center point of the face and the direction vector of the face relative to the target. The angle between the head direction and the target direction is calculated based on the three-dimensional posture parameters. The weighted average model is constructed based on the pixel motion field and the angle between the head direction and the target direction.

8. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 7, characterized in that, The expression for the weighted average model is: ;in, The motion intensity of pixels in adjacent frames. For pixel sports fields, For instantaneous correlation score, For instantaneous time index, For time window, As the first adjustment factor, The correlation score is weighted over time.

9. The method for extracting key head poses of multiple targets based on lookahead guidance according to claim 8, characterized in that, The aligned image is filtered for key targets based on the weighted average model to obtain key head pose results, including: A comprehensive correlation index is calculated based on the direction of motion; the formula for calculating the comprehensive correlation index is: ;in, The rate of change of the target velocity in the direction of motion. The second adjustment factor; The three-dimensional pose parameter with the highest score among the comprehensive relevant indicators is selected as the key head pose result.

Images