End-to-end multi-person head pose estimation method and device based on 6D rotation representation

Abstract

The present invention discloses an end-to-end multi-person head posture estimation method and device based on 6D rotation representation, the method steps include: obtaining an image training set and inputting it into a multi-person head posture detection model for training, during the training process, firstly extracting features from the input image, predicting the head prediction parameters of multiple head objects and the head posture 6D rotation representation parameters, converting the two three-dimensional head posture rotation matrices of each head object to obtain a head posture estimation matrix; continuously adjusting the parameters of the multi-person head posture detection model according to the similarity between the real head posture matrix of each head object and the head posture estimation matrix until a trained multi-person head posture detection model is obtained; obtaining an image to be tested, inputting the image to be tested into the trained multi-person head posture detection model, and obtaining detection results of multiple head objects. The present invention can improve the end-to-end multi-person head posture estimation accuracy while maintaining the real-time detection.

Description

Technical Field

[0001] The present invention relates to the technical field of posture estimation, and in particular to an end-to-end multi-person head posture estimation method and device based on 6D (Dimension) rotation representation. Background Art

[0002] Head posture is an important factor in judging human behavior and is also very helpful in understanding human behavior. Head posture estimation refers to the use of computer technology to analyze and predict input images or video sequences to determine the position and posture parameters of a person's head in three-dimensional space. Head posture estimation technology has a wide range of applications in human-computer interaction, biometrics, virtual reality, and fatigue detection.

[0003] For head pose estimation, most of the current solutions are aimed at single-person head pose estimation, that is, they can only realize the pose estimation of a single head object. For application scenarios that require simultaneous head pose estimation of multiple people, directly using this type of single-person head pose estimation method to realize multi-person head pose estimation not only has low detection efficiency, but also is complex to implement, which is not conducive to actual engineering deployment and application.

[0004] Although some practitioners have proposed using target detection algorithms to estimate the head posture of multiple people, by adding the Euler angle detection branch to the target detection algorithm to optimize the parameters of the head posture prediction branch, it is possible to achieve the training and recognition of multiple head posture recognition models. However, using Euler angles to directly predict head posture will result in increasing cumulative errors with the increase of continuous rotations, the rotation represented by Euler angles is not unique, which makes the process of restoring the rotation matrix by Euler angles complicated, and the computational complexity is high, resulting in insufficient accuracy of posture estimation. Summary of the invention

[0005] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides an end-to-end multi-person head posture estimation method and device based on 6D rotation representation with simple implementation method, high estimation accuracy and strong real-time performance, which can improve the end-to-end multi-person head posture estimation accuracy while maintaining the real-time performance of detection.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] An end-to-end multi-person head pose estimation method based on 6D rotation representation, comprising the following steps:

[0008] Obtaining an image training set and inputting it into a neural network-based multi-person head posture detection model for training, wherein the images in the image training set include multiple head objects;

[0009] During the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores, head detection frame position parameters, and head posture estimation parameters, and the head posture estimation parameters include head posture 6D rotation representation parameters of each head object predicted based on the head posture estimation branch; the head posture 6D rotation representation parameters of each head object are converted to obtain a head posture estimation matrix; the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches the preset requirement, and the trained multi-person head posture detection model is obtained;

[0010] The image to be tested is obtained, and the image to be tested is input into the trained multi-person head posture detection model to obtain the detection results of multiple head objects.

[0011] Furthermore, the head prediction score is a probability value of the current prediction being a head, and the head detection frame position parameters include the center position coordinates of the current predicted head detection frame and the width and height of the current predicted head detection frame.

[0012] Furthermore, the basic detection network of the multi-person head pose detection model is YOLOV (You Only LookOnce) 9, and a head pose estimation 6D decoupled detection head is set in the detection network based on YOLOV9 to predict the head pose 6D rotation representation parameters of the head object.

[0013] Further, the step of converting the head pose 6D rotation representation parameters of each head object into a head pose estimation matrix comprises: converting two unrelated three-dimensional vectors corresponding to the head pose 6D rotation representation parameters of each head object into u , v Convert to three mutually orthogonal unit vectors , , , each unit vector obtained by the transformation , , Construct the head posture estimation matrix, where:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, , , Respectively represent the 6D rotation parameter matrix Transformed into three three-dimensional unit vectors in a 3×3 rotation matrix, , , , ~ Respectively , , Each element in They are u exist x , y , z The coordinate values ​​of the axis, They are v exist x , y , z The coordinate values ​​of the axis, represents the head pose estimation matrix, Representation and A vector that is perpendicular in three-dimensional space.

[0022] Furthermore, the step of continuously adjusting the parameters of the multi-person head posture detection model according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix includes:

[0023] Get the real head posture matrix composed of real head posture vectors :

[0024]

[0025] in, , , are three mutually orthogonal unit vectors of the true head posture;

[0026] Using a comprehensive loss function to measure the similarity between the real head posture matrix and the head posture estimation matrix, the comprehensive loss function is weighted by using the loss function of the head prediction probability detection branch in the corresponding multi-person head posture detection model, the loss function of the head detection frame detection branch in the corresponding multi-person head posture detection model, and the loss function of the head posture estimation parameter detection branch in the corresponding multi-person head posture detection model;

[0027] Adjusting the parameters of the multi-person head posture detection model according to the value of the comprehensive loss function during the training process;

[0028] Use the adjusted multi-person head pose detection model to re-detect the input image to obtain a new head pose rotation matrix, and return to retrain the model using the comprehensive loss function until the similarity between the real head pose matrix and the head pose estimation matrix reaches the preset condition.

[0029] Furthermore, the calculation expression of the comprehensive loss function used in the process of training the multi-person head posture detection model is:

[0030]

[0031]

[0032]

[0033] in, Indicates the number of data samples processed at one time by the multi-person head posture detection model during training. , , Respectively , and The weight of the three, Represents the loss function of the head detection box branch, represents the loss function of the head prediction probability branch, represents the loss function of the parameter branch of head pose estimation, Represents the head posture matrix in i The loss weight on the dimension, , , represents three mutually orthogonal head pose estimation unit vectors in the head pose estimation matrix, Indicates i The head probability predicted by the head detection box, Indicates i The real head label of the head detection box, Indicates i The overlap between the head detection frame and the real head frame, BCE represents binary cross entropy, and N represents the total number of head detection frames in the head detection picture.

[0034] Furthermore, the loss function of the head detection box branch is The calculation expression is:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040] = +1

[0041] = -

[0042] =1-

[0043] in, , Respectively , The weight of Indicates the loss value of the center coordinates and frame length of the head detection frame. is the DFL loss function, i Indicates the sequence number of the head detection frame in the head detection picture. express No. j Quantity, represents the predicted value of the i-th head detection position in the picture, and M represents The total dimensions of Indicates the left boundary label value of the i-th head position in the image, , Indicates the right boundary label value of the i-th head position in the image, represents the left boundary weight of the i-th head position in the picture, represents the right boundary weight of the i-th head position in the picture, Indicates i The anchor point position corresponding to the head detection box, Indicates i The starting point coordinates of the real position of the head detection frame, Represents the parameters of the i-th head detection box in the image.

[0044] Furthermore, after obtaining the detection results of multiple head objects using the multi-person head posture detection model, a post-data processing step is also included, and the steps include:

[0045] Filter out detection frames whose confidence is less than a preset threshold from the detection results of multiple head objects currently obtained, and obtain the detection results after filtering;

[0046] Calculating the proximity between all detection frames in the detection result after the filtering process, and classifying all detection frames according to the proximity, wherein the detection frames with proximity less than a preset threshold are grouped into one cluster;

[0047] Calculate the confidence of each cluster after classification and use the confidence to weight the proximity between each pair of detection boxes;

[0048] Find the detection box with the smallest confidence-weighted proximity in each cluster and use it as the detection result of each cluster;

[0049] Detection frames whose proximity is less than a preset threshold are filtered out from the current detection result until the final head object detection result is obtained.

[0050] An end-to-end multi-person head pose estimation device based on 6D rotation representation, comprising:

[0051] A training module is used to obtain an image training set and input it into a multi-person head posture detection model for training, wherein the images in the image training set include multiple head objects; during the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores, head detection frame position parameters, and head posture estimation parameters, wherein the head posture estimation parameters include head posture 6D rotation representation parameters of each head object predicted based on the head posture estimation branch; the head posture 6D rotation representation parameters of each head object are converted to obtain a head posture estimation matrix; the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches a preset requirement, thereby obtaining a trained multi-person head posture detection model;

[0052] The real-time estimation module is used to obtain the image to be tested, input the image to be tested into the trained multi-person head posture detection model, and obtain the detection results of multiple head objects.

[0053] A computer device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0054] Compared with the prior art, the advantages of the present invention are: the present invention constructs an end-to-end multi-person head posture detection model based on a neural network, and at the same time sets a head posture estimation branch based on 6D rotation representation in the model to estimate the 6D rotation representation parameters of the head posture, and converts the three-dimensional head posture rotation matrix composed of the 6D rotation representation parameters of the head posture to obtain the head posture estimation matrix. During the training process of the model, the model parameters are continuously optimized and adjusted according to the similarity between the real head posture matrix and the head posture estimation matrix, so that an accurate multi-person head posture detection model can be quickly obtained, the accuracy of end-to-end multi-person head posture estimation is improved, and the problem of inaccurate posture estimation caused by cumulative errors, non-unique representation, high computational complexity, etc. in the traditional multi-person head posture estimation based on Euler angles is solved. At the same time, the head posture recognition algorithm model training and recognition process can be simplified, thereby improving the real-time performance of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the implementation flow of the end-to-end multi-person head posture estimation method based on 6D rotation representation in this embodiment.

[0056] Figure 2 It is a schematic diagram of the principle of realizing multi-person head posture estimation based on YOLOV9 in this embodiment.

[0057] Figure 3 It is a schematic diagram of the implementation flow of post-data processing based on the non-maximum suppression method that does not depend on confidence and Iou in this embodiment. DETAILED DESCRIPTION

[0058] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0059] like Figure 1 As shown, the steps of the end-to-end multi-person head posture estimation method based on 6D rotation representation in this embodiment include:

[0060] Step S01. Obtain an image training set and input it into a multi-person head posture detection model based on a neural network for training, wherein the images in the image training set include multiple head objects; during the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores, head detection frame position parameters, and head posture estimation parameters, wherein the head posture estimation parameters include head posture 6D rotation representation parameters of each head object predicted based on the head posture estimation branch; the head posture 6D rotation representation parameters of each head object are converted to obtain a head posture estimation matrix; and the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches a preset requirement, thereby obtaining a trained multi-person head posture detection model.

[0061] Step S02: Acquire the image to be tested, input the image to be tested into the trained multi-person head posture detection model, and obtain detection results of multiple head objects.

[0062] Embedding head position detection and head posture estimation into a feature representation layer is conducive to network learning of the intrinsic connection between the two, and using a six-dimensional rotation representation method can realize a continuous rotation matrix, which is conducive to neural network learning. This embodiment constructs an end-to-end multi-person head posture detection model based on a neural network, and at the same time sets a head posture estimation branch based on 6D rotation representation in the model to estimate the 6D rotation representation parameters of the head posture, and uses the three-dimensional vector transformation composed of the 6D rotation representation parameters of the head posture to obtain a head posture estimation matrix instead of Euler angles. During the training process of the model, the model parameters are continuously optimized and adjusted according to the similarity between the real head posture matrix and the head posture estimation matrix to quickly obtain an accurate multi-person head posture detection model, improve the accuracy of end-to-end one-stage multi-person head posture estimation, and solve the problem of inaccurate posture estimation caused by discontinuous rotation representation methods when traditional multi-person head posture estimation based on Euler angles is realized, thereby improving the head posture accuracy while maintaining a certain detection real-time performance, which is convenient for actual engineering deployment and application.

[0063] In this embodiment, the head prediction score is specifically the probability value of the current prediction being a head, and the head detection frame position parameters include the center position coordinates of the current predicted head detection frame and the width and height of the current predicted head detection frame. It can be understood that in addition to the above parameters, the head prediction score can also use other parameters related to the prediction score, and the head detection frame position parameters can also use other parameters related to the detection frame position. The specific parameter type can be selected and configured according to actual needs.

[0064] For example, a multi-person head pose detection model can use a dense detection network to directly predict a set of head objects. ,in , Represents the head prediction score and position parameters, Represents the 6D rotation representation parameters of the head pose; The specific parameters are , Represents the probability value of the current prediction being the head, Represents the center position of the current predicted head detection box, Represents the width and height of the current predicted head detection box; The specific parameters are , represents the 6D rotation representation parameter, These 6 values ​​form two three-dimensional vectors: , , u , v They are the three-dimensional head posture rotation matrix, They are the three-dimensional head posture rotation matrices u exist x , y , z The coordinate values ​​of the axis, They are the three-dimensional head posture rotation matrices v exist x , y , z The coordinate values ​​of the axis, u , v are two non-parallel vectors. As an attribute of the head object, compare it with Combine to form the head prediction object , it is possible to achieve end-to-end one-stage multi-person head pose high-precision estimation based on 6D rotation representation.

[0065] The head pose 6D rotation representation parameters are composed of two unrelated vectors. In order to convert the head pose 6D rotation representation parameters of each head object into a head pose estimation matrix, this embodiment converts the head pose 6D rotation representation parameters into three mutually orthogonal unit vectors, and then converts the three mutually orthogonal unit vectors to obtain the head pose estimation matrix, so that the head pose estimation matrix can be converted based on the head pose 6D rotation representation parameters.

[0066] In this embodiment, the 6D rotation representation parameter is based on the head posture. , 3D head pose rotation matrix u , v Convert to head pose estimation matrix The specific calculation expression is:

[0067] (1)

[0068] (2)

[0069] (3)

[0070] (4)

[0071] (5)

[0072] in, , , Respectively represent the 6D rotation parameter matrix Transform three 3D unit vectors into a 3×3 rotation matrix , , , , , Elements ~ The corresponding parameter is the 6D rotation representation of the head pose. Representation and A vector that is perpendicular in three-dimensional space.

[0073] As shown in the above equations (1), (2) and (4), Is the three-dimensional vector The unit vector obtained by normalization is is the unit vector obtained by normalizing the three-dimensional vector w. is with , The unit vector that is perpendicular to the plane of space.

[0074] The detection network is used to detect the 6D rotation representation parameters in the multi-person head posture detection model. Then, according to the above formulas (1) to (5), it can be converted into a 3×3 rotation matrix, which is the head posture estimation matrix Since the head posture estimation parameters adopt the 6D rotation representation method, this embodiment can further realize the representation of the continuous rotation matrix by converting the 6D rotation representation method into a 3×3 continuous rotation matrix in the above manner, so that the head posture estimation matrix can be directly used in the model training process. Optimizing the model parameters based on the distance between the real head pose estimation matrix is ​​beneficial to improving the accuracy of model training and thus improving the accuracy of head pose estimation.

[0075] In this embodiment, the multi-person head posture detection model is specifically constructed based on the detection network based on YOLOV9, and a head posture estimation 6D decoupling detection head is set in the detection network based on YOLOV9 to predict the 6D rotation representation parameters of the head posture of the head object.

[0076] When input data undergoes layer-by-layer feature extraction and spatial transformation in a deep learning model, a large amount of information will be lost. In the multi-person head detection scenario, since the head area belongs to a small or medium-sized target area, after the head area undergoes layer-by-layer feature extraction and spatial transformation by a deep learning model, a large amount of information loss will occur in the head area, which is not conducive to head area detection and posture estimation. YOLOv9 not only inherits the single-stage detection characteristics of the YOLO series of algorithms, but also solves the problem of data information loss during deep network transmission (i.e., information bottleneck and reversible function) through the programmable gradient information (PGI) and general efficient layer aggregation network (GELAN) architecture. PGI can provide complete input information for the target task to calculate the target function, reduce the information loss of data during deep network transmission, and thus obtain reliable gradient information to update the network weights. As a general efficient layer aggregation network that takes into account lightweight, inference speed and accuracy, the GELAN network can ensure high accuracy and real-time model detection. This embodiment takes into account the above-mentioned problems of multi-person head posture estimation and the characteristics of YOLOv9, and adopts YOLOv9 as the basic detection network for multi-person head posture estimation to give full play to the advantages of YOLOv9, greatly reduce the information loss of the head area during the multi-person head posture estimation process, and ensure the accuracy and real-time performance of the detection.

[0077] Specifically, the programmable gradient information PGI in YOLOv9 consists of three parts: the main branch, the auxiliary reversible branch (Auxiliary Reversible Branch) and the multi-level auxiliary information (Multi-level Auxiliary Information). The reasoning process of PGI only uses the main branch, so no additional reasoning cost is required. The auxiliary reversible branch is designed to deal with the problem that the loss function cannot generate reliable gradients due to the information bottleneck caused by the deepening of the neural network. Multi-level auxiliary information is designed to deal with the error accumulation problem caused by deep supervision, especially for architectures with multiple prediction branches and lightweight models.

[0078] The GELAN network adopts the concepts of segmentation and reorganization of CSPNet (Cross Stage Partial Network) and introduces the hierarchical convolution processing method of ELAN in each part. GELAN not only uses convolutional layers, but also can use any computing blocks to make the network more flexible and customized according to different application requirements. The design of GELAN takes into account lightweight, inference speed and accuracy to improve the overall performance of the model. The optionality of modules and partitions further increases the adaptability and customizability of the network, allowing the GELAN structure to support multiple types of computing blocks, so that it can better adapt to a variety of different computing needs and hardware constraints, and can also provide a more general and efficient network that can adapt to light to complex deep learning tasks while maintaining or enhancing computing efficiency and performance.

[0079] In this embodiment, 6 (configurable) prediction heads are specifically set in the multi-person head posture detection model based on YOLOV9, that is, there are 6 prediction output positions, which correspond to the output of 6 head posture estimation grids. Each grid output includes 11 channels in total, including head prediction probability, position detection frame and head posture estimation parameters. After the model inputs an RGB image, it first passes through the feature extraction layer, and then outputs 6 head posture estimation grids from the 6 prediction heads, and generates a dense head object o output, that is, .

[0080] like Figure 2 As shown, in this embodiment, in addition to the head prediction probability detection head (used to predict the classification probability Cls) and the head position detection frame detection head (used to predict the head position detection frame Bbox), a head posture estimation 6D decoupled detection head (Decoupled Head) is added on the basis of the YOLOv9 framework to predict the head posture 6D rotation representation parameters , that is, predicting the head pose. YOLOV9 is based on the anchor-free detection method. It uses a decoupled detection head to use sliding convolution in three ways to achieve category prediction (Cls), head position detection (Bbox) and head pose parameter (Head pose) estimation. The decoupled detection head method can speed up the model convergence and improve the model detection accuracy.

[0081] In this embodiment, the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, including:

[0082] Step S101. Obtain the real head posture matrix composed of real head posture vectors :

[0083] (6)

[0084] in, , , are three mutually orthogonal unit vectors of the true head posture;

[0085] Step S102. Use a comprehensive loss function to measure the similarity between the real head posture matrix and the head posture estimation matrix, where the comprehensive loss function is obtained by weighting the loss function of the head prediction probability detection branch in the corresponding multi-person head posture detection model, the loss function of the head detection frame detection branch in the corresponding multi-person head posture detection model, and the loss function of the head posture estimation parameter detection branch in the corresponding multi-person head posture detection model;

[0086] Step S103. Adjust the parameters of the multi-person head posture detection model according to the value of the comprehensive loss function during the training process;

[0087] Step S104. Use the adjusted multi-person head posture detection model to re-detect the input image to obtain a new head posture rotation matrix, and return to step S102 to re-train the model using the comprehensive loss function until the similarity between the real head posture matrix and the head posture estimation matrix reaches a preset condition.

[0088] The key to the deep learning method is to design a suitable objective function so that the prediction results of the model can be closest to the actual situation. In this embodiment, since the detection head of the multi-person head posture estimation model includes detection heads for head prediction probability, head detection frame and head posture estimation parameters respectively, the loss function used in the model training process is a comprehensive loss function obtained by integrating the loss functions of each detection branch, which can make the multi-person head prediction results closest to the actual situation and ensure the accuracy of the prediction.

[0089] In a specific application embodiment, the calculation expression of the comprehensive loss function used in the process of training the multi-person head posture detection model is:

[0090] (7)

[0091] in, Indicates the number of data samples (batch size) processed at one time by the multi-person head posture detection model during training. , , Respectively , and The weight of the three, Represents the loss function of the head detection box branch, represents the loss function of the head prediction probability branch, Represents the loss function for the detection head pose estimation parameter branch.

[0092] In order not to destroy the SO(3) manifold geometry structure of the head pose 3x3 rotation matrix, this embodiment uses a method based on cosine similarity to characterize the head pose estimation matrix With the real head pose matrix The similarity between them is used to construct the loss function for detecting the head posture estimation parameter branch. The detailed steps are:

[0093] The 6D branch predicts 6 values ​​from the detected head pose estimation parameter branch at each anchor point , forming two three-dimensional vectors: , , and then converted into a head posture estimation matrix according to the above formulas (1) to (6) , the matrix is ​​an orthogonal matrix;

[0094] The label rotation matrix obtained by Euler angle is also an orthogonal matrix:

[0095]

[0096] in, , , are three mutually orthogonal head true posture unit vectors. The above matrix That is the real head posture matrix .

[0097] In the multi-person head pose model, a loss function is needed to measure the head pose estimation matrix With the real head pose matrix The distance between them enables the model to continuously optimize parameters and predict the head posture rotation matrix. In order to calculate the head posture estimation matrix With the real head pose matrix The similarity / distance between them, the method based on cosine similarity in this embodiment adopts the following method to construct the loss function of the branch for detecting head posture estimation parameters: :

[0098] (8)

[0099] in, Represents the head posture matrix i The loss weight in dimension, , , Represents three mutually orthogonal head pose estimation unit vectors in the head pose estimation matrix.

[0100] This embodiment constructs the loss function of the head posture estimation parameter branch by the above method , will not destroy the SO(3) popular geometry structure of the 3x3 rotation matrix of the head pose, and because ,thus , that is, the loss function value range of the head posture estimation parameter branch is required to be greater than or equal to 0, while the arccosine function value range is greater than 0. The arccosine function can amplify the loss value when the posture estimation is inaccurate, which is beneficial for the model to focus on difficult sample learning. Therefore, by setting the adjustment factor The ratio between the three dimensions can be further adjusted dynamically to ensure the accuracy of head posture estimation and make the loss value approach 0. For example, if the similarity of a dimension component in the two matrices is small, the loss proportion contributed by it will be further increased. In this way, the multi-person head posture model can further focus on difficult samples, thereby further improving the accuracy of head posture estimation.

[0101] In this embodiment, the loss function of the head prediction probability branch is The specific calculation is as follows:

[0102] (9)

[0103] in, Indicates i The head probability predicted by the head detection box, Indicates i The real head label of the head detection box, Indicates i The overlap between the head detection frame and the real head frame, BCE represents binary cross entropy, and N represents the total number of head detection frames in the head detection picture.

[0104] In this embodiment, the loss function of the head detection frame branch is The calculation expression is:

[0105] (10)

[0106] (11)

[0107] (12)

[0108] (13)

[0109] (14)

[0110] = +1 (15)

[0111] = - (16)

[0112] =1- (17)

[0113] in, , Respectively , The weight of Indicates the loss value of 4 parameters (frame center point coordinates and frame length) in the head detection frame. The DFL loss function is used to correct the model's error in predicting the bounding box of an object during the bounding box regression process in the target detection task, so that it can optimize blurred or out-of-focus images to a certain extent and improve the accuracy of object detection. i Indicates the sequence number of the head detection frame in the head detection picture. express No. j Quantity, represents the predicted value of the i-th head detection position in the picture, and M represents The total dimensions of Indicates the left boundary label value of the i-th head position in the image, , Indicates the right boundary label value of the i-th head position in the image, represents the left boundary weight of the i-th head position in the picture, represents the right boundary weight of the i-th head position in the picture, Indicates i The anchor point position corresponding to the head detection box, Represents the parameters of the i-th head detection box in the image, Respectively represent i The horizontal and vertical coordinates of the starting point and the end point of the real position of the head detection frame, where subscript 1 corresponds to the starting point and subscript 2 corresponds to the end point. For a rectangular detection frame, the coordinate point of the upper left corner of the detection frame can be taken as the starting point and the coordinate point of the lower right corner can be taken as the end point.

[0114] This embodiment represents the head posture estimation parameters based on the 6D rotation representation method. The 6D rotation representation method is first converted into a 3x3 continuous rotation matrix, and then a specific loss function is constructed in this way to achieve the optimization and update of the head posture parameters. This can make the prediction results as close to the actual situation as possible and improve the accuracy of head posture estimation for multiple people.

[0115] In this embodiment, after obtaining the detection results of multiple head objects using the multi-person head posture detection model, a post-data processing step S03 is also included. Figure 3 As shown, the detailed steps include:

[0116] Step S301. Filter out detection frames whose confidence is less than a preset threshold from the detection results of multiple head objects currently obtained, and obtain the detection results after filtering;

[0117] Step S302: Calculate the proximity between all detection frames in the detection result after filtering, and classify all detection frames according to the proximity, wherein the detection frames with proximity less than a first preset threshold are grouped into one cluster;

[0118] Step S303. Calculate the confidence of each cluster after classification according to the Manhattan distance and use the confidence to weight the proximity P between each pair of detection boxes;

[0119] Step S304: Find the detection box with the smallest confidence-weighted proximity P in each cluster and use it as the detection result of each cluster;

[0120] Step S305. Filter out other detection frames whose proximity P is less than the second preset threshold from the current detection result to filter out redundant detection frames. For the remaining frames, repeat steps S302-S305 until the final head object detection result is obtained, and the second preset threshold is less than the first preset threshold.

[0121] After inference, the multi-person head posture detection model will generate a series of head detection objects, and this series of detection objects needs to be post-processed to obtain the final real detection object. Since the multi-person head posture estimation scene belongs to dense target detection, there are different degrees of overlap between targets. The traditional non-maximum suppression algorithm (nms) for post-processing of target posture detection only relies on the highest confidence of target classification and Iou (Intersection over Union), which leads to a certain degree of inaccurate positioning and missed detection for overlapping detection scenes. This embodiment classifies all head object detection frames according to the proximity between the detection frames, and then uses the confidence to weight the proximity, finds the smallest detection frame after confidence weighting to obtain the detection result, does not need to rely on confidence and Iou, can solve the problem that the traditional non-maximum suppression method must rely on confidence and Iou, so that the detection accuracy of overlapping scenes can be improved, thereby improving the accuracy of head posture estimation.

[0122] In a specific application embodiment, the relevant coordinates may be normalized first, and then the proximity P may be calculated using a method such as Manhattan distance.

[0123] The end-to-end multi-person head posture estimation device based on 6D rotation representation in this embodiment includes:

[0124] A training module is used to obtain an image training set and input it into a multi-person head posture detection model for training, wherein the images in the image training set include multiple head objects; during the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters and head posture 6D rotation representation parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores and head detection frame position parameters, and the head posture 6D rotation representation parameters form two three-dimensional head posture rotation matrices; the two three-dimensional head posture rotation matrices of each head object are converted to obtain a head posture estimation matrix; the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches the preset requirement, thereby obtaining a trained multi-person head posture detection model;

[0125] The real-time estimation module is used to obtain the image to be tested, input the image to be tested into the trained multi-person head posture detection model, and obtain the detection results of multiple head objects.

[0126] The end-to-end multi-person head pose estimation device based on 6D rotation representation in this embodiment corresponds one-to-one to the above-mentioned end-to-end multi-person head pose estimation method based on 6D rotation representation, and will not be described one by one here.

[0127] This embodiment further provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0128] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0129] Those skilled in the art should understand that the above-mentioned embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. An end-to-end multi-person head pose estimation method based on 6D rotation representation, characterized in that the steps include: Obtaining an image training set and inputting it into a neural network-based multi-person head posture detection model for training, wherein the images in the image training set include multiple head objects; During the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores, head detection frame position parameters, and head posture estimation parameters, and the head posture estimation parameters include head posture 6D rotation representation parameters of each head object predicted based on the head posture estimation branch; the head posture 6D rotation representation parameters of each head object are converted to obtain a head posture estimation matrix; the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches the preset requirement, and the trained multi-person head posture detection model is obtained; Obtaining an image to be tested, inputting the image to be tested into a trained multi-person head posture detection model to obtain detection results of multiple head objects; The method of continuously adjusting the parameters of the multi-person head posture detection model according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix includes: Get the real head posture matrix composed of real head posture vectors : in, , , are three mutually orthogonal unit vectors of the true head posture; Using a comprehensive loss function to measure the similarity between the real head posture matrix and the head posture estimation matrix, the comprehensive loss function is weighted by using the loss function of the head prediction probability detection branch in the corresponding multi-person head posture detection model, the loss function of the head detection frame detection branch in the corresponding multi-person head posture detection model, and the loss function of the head posture estimation parameter detection branch in the corresponding multi-person head posture detection model; Adjusting the parameters of the multi-person head posture detection model according to the value of the comprehensive loss function during the training process; Use the adjusted multi-person head pose detection model to re-detect the input image to obtain a new head pose rotation matrix, and return to re-train the model using the comprehensive loss function until the similarity between the real head pose matrix and the head pose estimation matrix reaches the preset condition; The calculation expression of the comprehensive loss function used in the process of training the multi-person head posture detection model is: in, Indicates the number of data samples processed at one time by the multi-person head posture detection model during training. , , Respectively , and The weight of the three, Represents the loss function of the head detection box branch, represents the loss function of the detection head prediction probability branch, represents the loss function of the parameter branch of head pose estimation, Represents the head posture matrix in i The loss weight on the dimension, , , represents three mutually orthogonal head pose estimation unit vectors in the head pose estimation matrix, , , are three mutually orthogonal unit vectors of the true head posture, Indicates i The head probability predicted by the head detection box, Indicates i The real head label of the head detection box, Indicates i The overlap between the head detection frame and the real head frame, BCE represents binary cross entropy, and N represents the total number of head detection frames in the head detection picture.

2. The end-to-end multi-person head pose estimation method based on 6D rotation representation according to claim 1, characterized in that: The head prediction score is a probability value of the current prediction being a head, and the head detection frame position parameters include the center position coordinates of the current predicted head detection frame and the width and height of the current predicted head detection frame.

3. The end-to-end multi-person head pose estimation method based on 6D rotation representation according to claim 1, characterized in that: The basic detection network of the multi-person head posture detection model is YOLOV9, and a head posture estimation 6D decoupling detection head is set in the detection network based on YOLOV9 to predict the head posture 6D rotation representation parameters of the head object.

4. The end-to-end multi-person head pose estimation method based on 6D rotation representation according to claim 1, characterized in that: The step of converting the head pose 6D rotation representation parameters of each head object to obtain a head pose estimation matrix comprises: converting two unrelated three-dimensional vectors corresponding to the head pose 6D rotation representation parameters of each head object u , v Converted into three mutually orthogonal unit vectors, the unit vectors obtained by the conversion constitute the head posture estimation matrix, and the calculation expression is: in, , , Respectively represent the 6D rotation parameter matrix Transformed into three three-dimensional unit vectors in a 3×3 rotation matrix, , , , ~ Respectively , , Each element in They are u exist x , y , z The coordinate values ​​of the axis, They are v exist x , y , z The coordinate values ​​of the axis, represents the head pose estimation matrix, Representation and A vector that is perpendicular in three-dimensional space.

5. The end-to-end multi-person head pose estimation method based on 6D rotation representation according to claim 1, characterized in that: Loss function for detecting the head detection box branch The calculation expression is: = +1 = - =1- in, , Respectively , The weight of Indicates the loss value of the center coordinates and frame length of the head detection frame. is the DFL loss function, i Indicates the sequence number of the head detection frame in the head detection picture. express No. j Quantity, represents the predicted value of the i-th head detection position in the picture, and M represents The total dimensions of Indicates the left boundary label value of the i-th head position in the image, , Indicates the right boundary label value of the i-th head position in the image, represents the left boundary weight of the i-th head position in the picture, represents the right boundary weight of the i-th head position in the picture, Indicates i The anchor point position corresponding to the head detection box, Indicates i The starting point coordinates of the real position of the head detection frame, Represents the parameters of the i-th head detection box in the image.

6. The end-to-end multi-person head pose estimation method based on 6D rotation representation according to any one of claims 1 to 5, characterized in that: After obtaining the detection results of multiple head objects using the multi-person head posture detection model, a post-data processing step is also included, which includes: Filter out detection frames whose confidence is less than a preset threshold from the detection results of multiple head objects currently obtained, and obtain the detection results after filtering; Calculating the proximity between all detection frames in the detection result after the filtering process, and classifying all detection frames according to the proximity, wherein the detection frames with proximity less than a preset threshold are grouped into one cluster; Calculate the confidence of each cluster after classification and use the confidence to weight the proximity between each pair of detection boxes; Find the detection box with the smallest confidence-weighted proximity in each cluster and use it as the detection result of each cluster; Detection frames whose proximity is less than a preset threshold are filtered out from the current detection result until the final head object detection result is obtained.

7. A device for implementing the end-to-end multi-person head pose estimation method based on 6D rotation representation as described in any one of claims 1 to 6, characterized in that: include: A training module, used for acquiring an image training set and inputting the image training set into a multi-person head posture detection model for training, wherein the image training set includes multiple head objects; During the training process, the multi-person head posture detection model first extracts features from the input image, and predicts head prediction parameters of multiple head objects based on the extracted features, wherein the head prediction parameters include head prediction scores, head detection frame position parameters, and head posture estimation parameters, and the head posture estimation parameters include head posture 6D rotation representation parameters of each head object predicted based on the head posture estimation branch; the head posture 6D rotation representation parameters of each head object are converted to obtain a head posture estimation matrix; the parameters of the multi-person head posture detection model are continuously adjusted according to the similarity between the real head posture matrix of each head object and the obtained head posture estimation matrix, until the similarity reaches the preset requirement, and the trained multi-person head posture detection model is obtained; The real-time estimation module is used to obtain the image to be tested, input the image to be tested into the trained multi-person head posture detection model, and obtain the detection results of multiple head objects.

8. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 6.

Images