Head posture recognition method, device and equipment and storage medium

Abstract

The invention relates to a head posture recognition method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring a to-be-processed head image; inputting a to-be-processed head image into the posture recognition model to obtain a rotation matrix; the posture recognition model is obtained based on loss function training of a regression algorithm; the loss function of the regression algorithm comprises a loss function of a rotation axis vector of logarithm mapping of the rotation matrix on the Lie algebra, and is used for reflecting an included angle between the rotation axis vector of logarithm mapping of a predicted rotation matrix obtained based on a regression algorithm training attitude recognition model and a real rotation matrix on the corresponding Lie algebra; the rotation matrix is used for representing the posture of the head in the to-be-processed head image in the three-dimensional space. Therefore, the precision and accuracy of the rotation matrix can be improved, and the error of the head posture recognition model is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of autonomous driving technology, and specifically to a head posture recognition method, device, equipment and storage medium. Background Art

[0002] In the field of autonomous driving, the relationship between the driver and the vehicle's autonomous driving system needs to be properly handled, especially when the driver shows abnormal behaviors such as fatigue or distraction, the vehicle's autonomous driving system needs to take over the vehicle. Among them, the main method of judging whether the driver has abnormal behavior is through head posture recognition. Traditional head posture recognition methods can only recognize the head posture of the front face or slightly side face, and then detect the key points of the face and use the key point alignment method to recognize the face posture. This method does not use an end-to-end model, and there are errors in each step. The accumulated errors lead to inaccurate final results. In addition, the detection of facial key points has high requirements for images. In poor lighting conditions, key points cannot be recognized.

[0003] In the related art, the residual network is simulated as a three-dimensional cube with multiple channels of a deep residual network with 34 layers (ResNet-34), and three vectors are determined by integrating the three dimensions to form a rotation matrix. In this method, the model is trained by the loss between the rotation matrix of the real label and the rotation matrix of the predicted value. The loss function will oscillate during the model training stage, and the degree of freedom of rotation in the three-dimensional space is 3, while the model needs to regress 9 numbers, which makes the degree of freedom of model regression too high, resulting in a large error in the rotation matrix, and the head posture in the image cannot be accurately represented. Therefore, how to reduce the error of the rotation matrix and improve the accuracy of the posture recognition model is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a head posture recognition method, device, equipment and storage medium to at least solve the technical problems in the related art that the rotation matrix has a large error and the posture recognition model has a low accuracy. The technical solution of the present application is as follows:

[0005] According to the first aspect of the present application, a head posture recognition method is provided, the method comprising: obtaining a head image to be processed; inputting the head image to be processed into a posture recognition model to obtain a rotation matrix; the posture recognition model is trained based on a loss function of a regression algorithm; the loss function of the regression algorithm comprises a loss function of a rotation axis vector of a logarithmic mapping of a rotation matrix on a Lie algebra, which is used to reflect the angle between a predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and a rotation axis vector of a logarithmic mapping of a true rotation matrix on a corresponding Lie algebra; the rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

[0006] According to the above technical means, the regression algorithm can provide an accurate three-dimensional rotation matrix. On the basis of the loss function of the regression algorithm, the loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra is added, which avoids the oscillation of the loss function of the regression algorithm in the training process in the related technology, reduces the error of the rotation matrix output by the posture recognition model, and improves the accuracy of the rotation matrix, so that the rotation matrix can more accurately represent the state of the head in the head image to be processed, so that the posture recognition model can more accurately identify the position, direction and rotation angle of the head in the space in the head image to be processed.

[0007] In a possible implementation, the regression algorithm is a 6D regression algorithm; the loss function of the regression algorithm also includes: a loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra, a loss function of the angle between two three-dimensional vectors output by the posture recognition model, and a mean square error loss function.

[0008] According to the above technical means, the posture recognition model can also be trained based on the loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra in the regression algorithm, the loss function of the angle between the two three-dimensional vectors output by the posture recognition model, and the mean square error loss function, thereby avoiding the poor convergence of the loss function of the regression algorithm during the training process and improving the accuracy and stability of the posture recognition model.

[0009] In another possible implementation, the posture recognition model is also trained based on the loss function of the Euler angle prediction algorithm; the loss function of the Euler angle prediction algorithm includes: a loss function of the integer part of the Euler angle and a loss function of the remainder part of the Euler angle.

[0010] According to the above technical means, the posture recognition model can also be assisted in training based on the loss function of the Euler angle prediction algorithm, which improves the convergence of the loss function of the posture recognition model during the training process, thereby improving the accuracy and stability of the posture recognition model. In addition, the encoding method of the Euler angle is modified, and the angle of the Euler angle is divided into an integer part and a remainder part. The posture recognition model is trained based on the loss function of the integer part of the Euler angle and the loss function of the remainder part of the Euler angle, respectively, avoiding the loss oscillation caused by the encoding method of the Euler angle, and improving the accuracy of the posture recognition model.

[0011] In another possible implementation, the method further includes: constructing an initial posture recognition model; obtaining multiple head image data and rotation matrices corresponding to the multiple head image data; using the multiple head image data as samples and the rotation matrices corresponding to the multiple head image data as sample labels to obtain multiple sample data; performing image enhancement processing on the multiple sample data to obtain training data; and training the initial posture recognition model based on the training data to obtain a posture recognition model.

[0012] According to the above technical means, multiple sample data are subjected to image enhancement processing, so that the posture recognition model can recognize images in different states, thereby enhancing the generalization ability of the posture recognition model. In addition, the initial posture recognition model is trained based on multiple sample data, so that the posture recognition model can learn the complex relationship between the head image and the rotation matrix, thereby improving the recognition accuracy of the posture recognition model.

[0013] In another possible implementation, the initial posture recognition model includes: a backbone network, a feature conversion layer and an output layer; constructing the initial posture recognition model includes: constructing a backbone network of the initial posture recognition model based on a re-parameterization of visual geometry group (RepVGG) network; constructing a feature conversion layer of the initial posture recognition model based on an average pooling layer; and constructing an output layer of the initial posture recognition model based on a regression algorithm and an Euler angle prediction algorithm.

[0014] According to the above technical means, the RepVGG network can optimize the network structure during the training and reasoning stages of the model, so that the posture recognition model can extract richer features while maintaining computational efficiency. The average pooling layer can reduce the spatial dimension of the image and retain important information. The RepVGG network is used as the backbone network of the initial posture recognition model, and the average pooling layer is used as the feature conversion layer of the initial posture recognition model. This reduces the number of parameters of the posture recognition model, reduces the demand for computing power of the posture recognition model, and ensures the accuracy of the posture recognition model, making the posture recognition model more lightweight.

[0015] In another possible implementation, the initial posture recognition model is trained based on training data to obtain the posture recognition model, including: based on the training data, respectively determining the first output data of the regression algorithm and the second output data of the Euler angle prediction algorithm in the initial posture recognition model; determining the first loss value of the regression algorithm based on the first output data; determining the second loss value of the Euler angle prediction algorithm based on the second output data; adjusting the initial posture recognition model based on the first loss value and the second loss value, and training the adjusted initial posture recognition model; constructing a posture recognition model based on the trained initial posture recognition model; the output layer of the initial posture recognition model includes: a regression branch and a Euler angle prediction branch; the output layer of the posture recognition model includes: a regression branch and / or a Euler angle prediction branch.

[0016] According to the above technical means, the posture recognition model can be trained based on the regression branch and the Euler angle prediction branch, so that the posture recognition model can recognize the rotation of the head and the Euler angle of the head, thereby improving the posture recognition model's ability to recognize complex postures. In addition, the initial posture recognition model is adjusted based on the loss value of the regression branch and the Euler angle prediction branch, thereby improving the recognition accuracy of the posture recognition model on key features. In addition, the output layer of the posture recognition model can selectively include a regression branch and / or an Euler angle prediction branch as needed, so that the posture recognition model can adapt to different application scenarios, thereby improving the flexibility of the posture recognition model.

[0017] In another possible implementation, based on the first loss value and the second loss value, the initial posture recognition model is adjusted, and the adjusted initial posture recognition model is trained, including: based on the first loss value and the second loss value, using an SGD optimizer to iteratively optimize the initial posture recognition model; when the number of iterations meets a first preset threshold or the initial posture recognition model converges, stopping the optimization of the initial posture recognition model, and training the optimized initial posture recognition model based on the training data.

[0018] According to the above technical means, the initial posture recognition model is iteratively optimized based on the first loss value and the second loss value, thereby improving the accuracy and stability of the posture recognition model. The initial posture recognition model is optimized using the SGD optimizer so that the learning rate and parameters can be adjusted according to specific problems to adapt to different training requirements. In addition, by monitoring the loss value and the number of iterations, the training is stopped when the posture recognition model converges, thereby avoiding resource waste.

[0019] In another possible implementation, the head image to be processed is input into a posture recognition model to obtain two three-dimensional vectors; the two three-dimensional vectors are used to represent the position information and direction information of the head in the head image to be processed; based on the directions of the two three-dimensional vectors, the two three-dimensional column vectors are converted into a first rotation matrix and a second rotation matrix; the first rotation matrix and the second rotation matrix are interpolated and orthogonally processed to obtain a rotation matrix.

[0020] According to the above technical means, the directions of the two three-dimensional vectors output by the posture recognition model are used as a reference, the two three-dimensional vectors are converted into a first rotation matrix and a second rotation matrix, and the rotation matrix is ​​obtained through interpolation orthogonal processing. The interpolation processing can reduce the error caused by image noise or imperfect posture estimation. Through interpolation processing, the first rotation matrix and the second rotation matrix are smoothly transitioned to obtain the rotation matrix, so that the rotation matrix can more accurately represent the posture of the head in the head image to be processed. In addition, through interpolation processing, the need for high-frequency updates can be reduced, the computational burden can be reduced, and the processing speed can be improved.

[0021] In another possible implementation, the method further includes: determining a rotation axis vector of a logarithmic mapping of a rotation matrix on a Lie algebra; and determining whether the head posture in the head image to be processed is in an abnormal state based at least on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0022] According to the above technical means, the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra is used to determine whether the head posture in the head image to be processed is in an abnormal state, so as to issue a corresponding alarm message when the head posture in the head image to be processed is in an abnormal state.

[0023] In another possible implementation, the method further includes: determining the logarithmic mapping angle of the rotation matrix on the Lie algebra; determining whether the head posture in the head image to be processed is in an abnormal state based at least on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, including: determining whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle and the rotation axis vector of the rotation matrix on the Lie algebra.

[0024] According to the above technical means, it is also possible to determine whether the head posture in the head image to be processed is in an abnormal state through the angle of the logarithmic mapping of the rotation matrix on the Lie algebra and the rotation axis vector, so as to issue a corresponding alarm message when the head posture in the head image to be processed is in an abnormal state.

[0025] In another possible implementation, determining whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle and the rotation axis vector of the rotation matrix on the Lie algebra includes: determining the target logarithmic mapping angle and the target rotation axis vector required to rotate from the standard normal posture to the head posture in the head image to be processed based on the logarithmic mapping angle and the rotation axis vector of the rotation matrix on the Lie algebra; determining that the head posture of the head image to be processed is an abnormal posture when at least one of the following conditions is met: the target logarithmic mapping angle is within a first angle range, the target logarithmic mapping angle is outside a second angle range, the target rotation axis vector is within a first vector range, and the target rotation axis vector is outside a second vector range.

[0026] According to the above technical means, the target logarithmic mapping angle and the target rotation axis vector are compared with the first angle range, the second angle range, the first vector range, and the second vector range respectively to determine whether the head posture in the head image to be processed is abnormal, so that abnormal behavior can be discovered in time, an alarm can be issued, and corresponding safety measures can be taken to reduce the occurrence of accidents.

[0027] In another possible implementation, based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector, determining whether the head posture in the head image to be processed is in an abnormal state includes: when the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold, determining that the head posture of the head image to be processed is an abnormal posture.

[0028] According to the above technical means, the logarithmic mapping angle and rotation axis vector of the rotation matrix on Lie algebra are compared with the logarithmic mapping angle and rotation axis vector of the standard normal posture respectively to determine whether the head posture is abnormal, so that abnormal behavior can be discovered in time, an alarm can be issued and corresponding safety measures can be taken to reduce the occurrence of accidents.

[0029] In another possible implementation, the head image to be processed is a head image of a vehicle driver, or a head image of a pedestrian within the image acquisition range of the vehicle; the method also includes: determining the ratio of head images to be processed with abnormal postures among multiple head images to be processed within a preset period; when the ratio is greater than a fourth preset threshold, issuing an alarm message; the alarm message is used to notify the vehicle's automatic driving system to take over the vehicle and issue an alarm message for dangerous driving.

[0030] According to the above technical means, by monitoring and identifying the head posture of the driver or pedestrian, abnormal behavior of the driver or pedestrian can be detected in time, such as the driver falling asleep or the pedestrian suddenly entering the vehicle's driving path, and an alarm is issued to reduce the risk of accidents. In addition, when an abnormal posture is detected, the vehicle's automatic driving system is notified to take over the vehicle so that the automatic driving system can take corresponding measures, such as slowing down, stopping or changing the driving route, to avoid accidents and improve the safety of the vehicle.

[0031] According to the second aspect provided by the present application, a head posture recognition device is provided, which includes: an acquisition module and a processing module; the acquisition module is used to acquire a head image to be processed; the processing module is used to input the head image to be processed into a posture recognition model to obtain a rotation matrix; the posture recognition model is trained based on the loss function of a regression algorithm; the loss function of the regression algorithm includes a loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, which is used to reflect the angle between the predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and the rotation axis vector of the logarithmic mapping of the true rotation matrix on the corresponding Lie algebra; the rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

[0032] In a possible implementation, the regression algorithm is a 6D regression algorithm; the loss function of the regression algorithm also includes: a loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra, a loss function of the angle between two three-dimensional vectors output by the posture recognition model, and a mean square error loss function.

[0033] In another possible implementation, the posture recognition model is also trained based on the loss function of the Euler angle prediction algorithm; the loss function of the Euler angle prediction algorithm includes: a loss function of the integer part of the Euler angle and a loss function of the remainder part of the Euler angle.

[0034] In another possible implementation, the device further includes: a construction module and a training module. The construction module is used to construct an initial posture recognition model; the acquisition module is also used to acquire multiple head image data and rotation matrices corresponding to the multiple head image data; the processing module is also used to use the multiple head image data as samples and the rotation matrices corresponding to the multiple head image data as sample labels to obtain multiple sample data; the processing module is also used to perform image enhancement processing on the multiple sample data to obtain training data; the training module is used to train the initial posture recognition model based on the training data to obtain a posture recognition model.

[0035] In another possible implementation, the initial posture recognition model includes: a backbone network, a feature conversion layer and an output layer; a construction module, specifically used to construct the backbone network of the initial posture recognition model based on the RepVGG network; based on the average pooling layer, construct the feature conversion layer of the initial posture recognition model; based on the regression algorithm and the Euler angle prediction algorithm, construct the output layer of the initial posture recognition model.

[0036] In another possible implementation, the training module is specifically used to determine, based on the training data, the first output data of the regression algorithm in the initial posture recognition model and the second output data of the Euler angle prediction algorithm; determine the first loss value of the regression algorithm based on the first output data; determine the second loss value of the Euler angle prediction algorithm based on the second output data; adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model; construct a posture recognition model based on the trained initial posture recognition model; the output layer of the initial posture recognition model includes: a regression branch and a Euler angle prediction branch; the output layer of the posture recognition model includes: a regression branch and / or a Euler angle prediction branch.

[0037] In another possible implementation, the training module is specifically used to iteratively optimize the initial posture recognition model using an SGD optimizer based on the first loss value and the second loss value; when the number of iterations meets a first preset threshold or the initial posture recognition model converges, the optimization of the initial posture recognition model is stopped, and the optimized initial posture recognition model is trained based on the training data.

[0038] In another possible implementation, the processing module is specifically used to input the head image to be processed into a posture recognition model to obtain two three-dimensional vectors; the two three-dimensional vectors are used to represent the position information and direction information of the head in the head image to be processed; based on the directions of the two three-dimensional vectors, the two three-dimensional column vectors are converted into a first rotation matrix and a second rotation matrix; and the first rotation matrix and the second rotation matrix are interpolated and orthogonally processed to obtain the rotation matrix.

[0039] In another possible implementation, the apparatus further includes: a determination module. The determination module is used to determine the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra; the determination module is also used to determine whether the head posture in the head image to be processed is in an abnormal state based on at least the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0040] In another possible implementation, the determination module is also used to determine the logarithmic mapping angle of the rotation matrix on the Lie algebra; the determination module is specifically used to determine whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector.

[0041] In another possible implementation, the determination module is specifically used to determine the target logarithmic mapping angle and target rotation axis vector required to rotate from a standard normal posture to the head posture in the head image to be processed based on the angle of the logarithmic mapping of the rotation matrix on the Lie algebra and the rotation axis vector; when at least one of the following conditions is met, the head posture of the head image to be processed is determined to be an abnormal posture: the target logarithmic mapping angle is within a first angle range, the target logarithmic mapping angle is outside a second angle range, the target rotation axis vector is within a first vector range, and the target rotation axis vector is outside a second vector range.

[0042] In another possible implementation, the determination module is specifically used to determine that the head posture of the head image to be processed is an abnormal posture when the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold.

[0043] In another possible implementation, the head image to be processed is a head image of a vehicle driver, or a head image of a pedestrian within the image acquisition range of the vehicle; the above device further includes: an alarm module. The determination module is further used to determine the ratio of the head images to be processed with abnormal postures in a plurality of head images to be processed within a preset period; the alarm module is used to issue an alarm message when the ratio is greater than a fourth preset threshold; the alarm message is used to notify the vehicle's automatic driving system to take over the vehicle and issue an alarm message of dangerous driving.

[0044] According to the third aspect provided by the present application, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0045] According to the fourth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method in the above-mentioned first aspect and any possible implementation method thereof.

[0046] According to the fifth aspect provided by the present application, a computer program product is provided, the computer program product comprising computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0047] Therefore, the above technical features of the present application have the following beneficial effects:

[0048] (1) The regression algorithm can provide an accurate three-dimensional rotation matrix. On the basis of the loss function of the regression algorithm, a loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra is added, which avoids the oscillation of the loss function of the regression algorithm in the training process in the related technology, reduces the error of the rotation matrix output by the posture recognition model, and improves the accuracy of the rotation matrix, so that the rotation matrix can more accurately represent the state of the head in the head image to be processed, so that the posture recognition model can more accurately identify the position, direction and rotation angle of the head in the space in the head image to be processed.

[0049] (2) The posture recognition model can also be trained based on the loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra in the regression algorithm, the loss function of the angle between the two three-dimensional vectors output by the posture recognition model, and the mean square error loss function, thereby avoiding the poor convergence of the loss function of the regression algorithm during the training process and improving the accuracy and stability of the posture recognition model.

[0050] (3) The posture recognition model can also be assisted in training based on the loss function of the Euler angle prediction algorithm, thereby improving the convergence of the loss function of the posture recognition model during the training process, thereby improving the accuracy and stability of the posture recognition model. In addition, the encoding method of the Euler angle is modified, and the angle of the Euler angle is divided into an integer part and a remainder part. The posture recognition model is trained based on the loss function of the integer part of the Euler angle and the loss function of the remainder part of the Euler angle, respectively, avoiding the loss oscillation caused by the encoding method of the Euler angle, and improving the accuracy of the posture recognition model.

[0051] (4) The multiple sample data are subjected to image enhancement processing, so that the posture recognition model can recognize images of different states, thereby enhancing the generalization ability of the posture recognition model. In addition, the initial posture recognition model is trained based on the multiple sample data, so that the posture recognition model can learn the complex relationship between the head image and the rotation matrix, thereby improving the recognition accuracy of the posture recognition model.

[0052] (5) The RepVGG network can optimize the network structure during the training and inference stages of the model, so that the posture recognition model can extract richer features while maintaining computational efficiency. The average pooling layer can reduce the spatial dimension of the image and retain important information. The RepVGG network is used as the backbone network of the initial posture recognition model, and the average pooling layer is used as the feature conversion layer of the initial posture recognition model. This reduces the number of parameters of the posture recognition model, reduces the computing power required by the posture recognition model, and ensures the accuracy of the posture recognition model, making the posture recognition model more lightweight.

[0053] (6) The posture recognition model can be trained based on the regression branch and the Euler angle prediction branch, so that the posture recognition model can recognize the rotation of the head and the Euler angle of the head, thereby improving the posture recognition model's ability to recognize complex postures. In addition, the initial posture recognition model is adjusted based on the loss values ​​of the regression branch and the Euler angle prediction branch, thereby improving the recognition accuracy of the posture recognition model on key features. In addition, the output layer of the posture recognition model can selectively include the regression branch and / or the Euler angle prediction branch as needed, so that the posture recognition model can adapt to different application scenarios, thereby improving the flexibility of the posture recognition model.

[0054] (7) The initial posture recognition model is iteratively optimized based on the first loss value and the second loss value, thereby improving the accuracy and stability of the posture recognition model. The initial posture recognition model is optimized using the SGD optimizer so that the learning rate and parameters can be adjusted according to the specific problem to meet different training requirements. In addition, by monitoring the loss value and the number of iterations, the training is stopped when the posture recognition model converges, thereby avoiding resource waste.

[0055] (8) Based on the directions of the two three-dimensional vectors output by the posture recognition model, the two three-dimensional vectors are converted into a first rotation matrix and a second rotation matrix, and the rotation matrix is ​​obtained through interpolation orthogonal processing. The interpolation processing can reduce the error caused by image noise or imperfect posture estimation. Through the interpolation processing, the first rotation matrix and the second rotation matrix are smoothly transitioned to obtain the rotation matrix, so that the rotation matrix can more accurately represent the posture of the head in the head image to be processed. In addition, through the interpolation processing, the need for high-frequency updates can be reduced, the computational burden can be reduced, and the processing speed can be improved.

[0056] (9) By using the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, it is determined whether the head posture in the head image to be processed is in an abnormal state, so as to issue a corresponding alarm message when the head posture in the head image to be processed is in an abnormal state.

[0057] (10) It is also possible to determine whether the head posture in the head image to be processed is in an abnormal state through the angle of the logarithmic mapping of the rotation matrix on the Lie algebra and the rotation axis vector, so that when the head posture in the head image to be processed is in an abnormal state, a corresponding alarm message is issued.

[0058] (11) The target logarithmic mapping angle and the target rotation axis vector are compared with the first angle range, the second angle range, the first vector range, and the second vector range, respectively, to determine whether the head posture in the head image to be processed is abnormal, so that abnormal behavior can be discovered in time, an alarm can be issued, and corresponding safety measures can be taken to reduce the occurrence of accidents.

[0059] (12) The logarithmic mapping angle and rotation axis vector of the rotation matrix on the Lie algebra are compared with the logarithmic mapping angle and rotation axis vector of the standard normal posture, respectively, to determine whether the head posture is abnormal, so that abnormal behavior can be discovered in time, an alarm can be issued, and corresponding safety measures can be taken to reduce the occurrence of accidents.

[0060] (13) By monitoring and identifying the head posture of the driver or pedestrian, abnormal behavior of the driver or pedestrian can be detected in time, such as when the driver falls asleep or a pedestrian suddenly enters the vehicle's driving path, and an alarm is issued, reducing the risk of accidents. In addition, when an abnormal posture is detected, the vehicle's automatic driving system is notified to take over the vehicle so that the automatic driving system can take corresponding measures, such as slowing down, stopping, or changing the driving route, to avoid accidents, thereby improving vehicle safety.

[0061] It should be noted that the technical effects brought about by any implementation method in the second to fifth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0062] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0064] Figure 1 is a flow chart of a head posture recognition method according to an exemplary embodiment;

[0065] Figure 2 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0066] Figure 3is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0067] Figure 4 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0068] Figure 5 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0069] Figure 6 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0070] Figure 7 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0071] Figure 8 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0072] Fig. 9 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0073] Fig.10 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0074] Fig.11 is a diagram showing an architecture of a gesture recognition model according to an exemplary embodiment;

[0075] Fig.12 is a schematic diagram of abnormal posture recognition according to an exemplary embodiment;

[0076] Fig.13 is a schematic diagram of a gesture recognition model structure according to an exemplary embodiment;

[0077] Fig.14 is a flow chart of another head posture recognition method according to an exemplary embodiment;

[0078] Fig.15 is a block diagram of a head posture recognition device according to an exemplary embodiment;

[0079] Fig.16 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0080] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0081] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0082] In the field of autonomous driving, the relationship between the driver and the vehicle's autonomous driving system needs to be properly handled, especially when the driver shows abnormal behaviors such as fatigue or distraction, the vehicle's autonomous driving system needs to take over the vehicle. Among them, the main method for judging whether the driver has abnormal behavior is through head posture recognition. Traditional head posture recognition methods can only recognize the head posture of the front face or slightly side face, and then detect the key points of the face and use the key point alignment method to recognize the face posture. This method does not use an end-to-end model, and there are errors in each step. The error accumulation leads to inaccurate final results. In addition, the detection of facial key points has high requirements for images. In the case of poor lighting conditions (for example: dark, reflective), there are few key points on the side face, which will affect the final effect and fail to recognize the key points. In addition, there are few solutions that can recognize the head of any posture. This method will fail in scenes such as the top of the head and the back of the head, which restricts the use scenario. It cannot be used when the driver falls asleep on the steering wheel, the driver lowers his head significantly, or the driver turns his head significantly to look outside the car.

[0083] In the related art, the WHENet model can be used for head posture recognition. However, the output of the WHENet model is the Euler angle, which has a universal lock problem, and the backbone network of the model is relatively large. The Euler angle is solved by a method similar to classification. The interval between categories is the minimum error of the model, and the interval is 3 degrees, which makes it difficult for the model error to be less than 3 degrees. In addition, the loss function is prone to oscillation during training, and the quantization error is large. The loss function of the model consists of regression loss and classification loss. The model divides the Euler angle into many categories and represents them with discrete numbers, but there is still a problem that the probability of representing an angle as multiple categories multiplied by the sum of the category index is the same number. Taking the code implementation during inference as an example, assuming that the yaw angle of the true label is 30 degrees, any two indexes in the vector of the yaw angle output of the model with a sum of 140, and the values ​​of the two indexes in the vector of the yaw angle output of the model are equal to 0.5 after softmax, can meet the conditions. There are more than 40 such index pairs, resulting in an unstable loss function of the WHENet model and poor model results. The average error of the WHENet model on the AFLW2000 data is 5.42 degrees.

[0084] In the related technology, the 360RepNet model can also be used for head posture recognition. The direct output of the 360DRepNet model is a 6-dimensional vector, that is, 2 3D vectors, which are converted into a rotation matrix through the ordinary Schmidt orthogonal method. The training of the model is to regress 6 numbers (i.e. 6D) composed of 2 3D vectors. The loss function of the model is the angle of the logarithmic mapping on the Lie algebra corresponding to the rotation matrix required for the real label to rotate to the predicted label. During the training process, the loss function of the model has poor convergence. At about 5 degrees, the model result will oscillate. Because the angle in the logarithmic mapping from Lie group to Lie algebra is a surjection, a rotation axis vector is required to uniquely determine it. The average error of the 360DRepNet model on AFLW2000 data is 5.31 degrees, and the average error on AFLW2000 data is 4.52 degrees.

[0085] In the related art, the residual network can also be simulated as a three-dimensional cube with multiple channels of ResNet-34, and three vectors are obtained by integrating the three axes respectively to form a rotation matrix. This method directly obtains the rotation matrix, and the model uses the loss L1 loss function, that is, directly determines the loss between the rotation matrix (9 numbers) of the true label and the rotation matrix (9 numbers) of the predicted value. The loss function will oscillate during the model training stage. The degree of freedom of the model regression of this method is too high. The degree of freedom of rotation in the three-dimensional space is only 3, and the model regresses 9 numbers. Moreover, it is too abstract to directly use the rotation matrix for judgment. The model is ResNet-34, which is too large, and the loss function design is too simple, so it is difficult to obtain a good model effect. Therefore, how to reduce the error of the rotation matrix and improve the accuracy of the posture recognition model is a technical problem that needs to be solved urgently.

[0086] In response to the above problems, the present application proposes a head posture recognition method, in which a regression algorithm can provide an accurate three-dimensional rotation matrix. On the basis of the loss function of the regression algorithm, a loss function of the rotation axis quantity of the logarithmic mapping of the rotation matrix on the Lie algebra is added, thereby avoiding the oscillation of the loss function of the regression algorithm in the training process in the related technology, reducing the error of the rotation matrix output by the posture recognition model, and improving the accuracy of the rotation matrix, so that the rotation matrix can more accurately represent the state of the head in the head image to be processed, so that the posture recognition model can more accurately identify the position, direction and rotation angle of the head in the space in the head image to be processed.

[0087] For ease of understanding, the head posture recognition method provided in this application is specifically introduced below with reference to the accompanying drawings.

[0088] Figure 1 is a flow chart of a head posture recognition method according to an exemplary embodiment. Figure 1 As shown, the head posture recognition method includes the following steps:

[0089] S101: Obtain a head image to be processed.

[0090] As a possible implementation manner, the recognition device may obtain a head image to be processed, so as to input the head image to be processed into a posture recognition model to obtain a corresponding rotation matrix.

[0091] S102: Input the head image to be processed into a posture recognition model to obtain a rotation matrix.

[0092] Among them, the posture recognition model is trained based on the loss function of the regression algorithm. The loss function of the regression algorithm includes the loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, which is used to reflect the angle between the predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and the rotation axis vector of the logarithmic mapping of the true rotation matrix on the corresponding Lie algebra. The rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

[0093] As a possible implementation, after the recognition device obtains the head image to be processed, the head image to be processed can be input into a posture recognition model to obtain a rotation matrix, so as to determine the posture of the head in the image to be processed based on the rotation matrix.

[0094] Exemplarily, after the recognition device inputs the head image to be processed into the posture recognition model, the posture recognition model can perform posture recognition through a regression algorithm based on the state of the head in the head image to be processed, and obtain two three-dimensional vectors representing the position information and direction information of the head in the head image to be processed, and transform the two three-dimensional vectors into a first rotation matrix and a second rotation matrix. The recognition device can interpolate the first rotation matrix and the second rotation matrix to obtain a rotation matrix.

[0095] It should be understood that the head posture recognition method of the present application proposes a new loss function in the regression algorithm, namely, the loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra. The loss function reflects the angle between the predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and the rotation axis vector of the logarithmic mapping of the real rotation matrix on the corresponding Lie algebra. The posture recognition model is trained based on the loss function, which can ensure that the adjustment of the rotation matrix during the training process is small and continuous, rather than sudden and jumping, so that the posture recognition model can smoothly adjust the head posture during the training process to avoid unnatural rotation changes. In addition, it ensures that the regression value of the rotation matrix and the regression value of the posture recognition model are in one-to-one correspondence, which satisfies the one-to-one mapping relationship between the rotation matrix and the Lie algebra, and avoids the situation where the loss of the posture recognition model oscillates during the training process, thereby improving the accuracy and stability of the posture recognition model.

[0096] In some embodiments, the regression algorithm is a 6D regression algorithm, and the loss function of the regression algorithm also includes: a loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra, a loss function of the angle between two three-dimensional vectors output by the posture recognition model, and a mean square error loss function. Therefore, the posture recognition model can also be trained based on the loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra in the regression algorithm, the loss function of the angle between two three-dimensional vectors output by the posture recognition model, and the mean square error loss function, thereby avoiding the situation where the loss of the posture recognition model has poor convergence during the training process, and improving the accuracy and stability of the posture recognition model. The embodiment of the present application takes the 6D regression algorithm as an example to explain the training process of the posture recognition model.

[0097] In some other embodiments, the posture recognition model is also trained based on the loss function of the Euler angle prediction algorithm, and the loss function of the Euler angle prediction algorithm includes: the loss function of the integer part of the angle of the Euler angle and the loss function of the remainder part of the angle of the Euler angle. Therefore, the posture recognition model can be trained not only based on the loss function of the regression algorithm, but also based on the Euler angle prediction algorithm. In addition, the head posture recognition method of the present application divides the angle of the Euler angle in the Euler angle prediction algorithm into an integer part and a remainder part, thereby training the posture recognition model based on the loss function of the integer part of the angle of the Euler angle and the loss function of the remainder part of the angle of the Euler angle, avoiding the loss oscillation caused by the encoding method of the angle of the Euler angle, improving the convergence of the loss function of the posture recognition model during the training process, thereby improving the accuracy and stability of the posture recognition model.

[0098] In some other embodiments, the recognition device may also construct an initial posture recognition model and train the initial posture recognition model based on the loss function of the regression algorithm and the loss function of the Euler angle prediction algorithm to obtain a posture recognition model. Figure 2 As shown, the head posture recognition method provided by the present application also includes the following steps S201-S205:

[0099] S201, constructing an initial posture recognition model.

[0100] As a possible implementation method, the recognition device can construct an initial posture recognition model based on the regression algorithm and the Euler angle prediction algorithm, so as to train the initial posture recognition model based on the loss function of the regression algorithm and the loss function of the Euler angle prediction algorithm to obtain the posture recognition model.

[0101] Exemplarily, the initial posture recognition model includes a backbone network, a feature conversion layer and an output layer. The recognition device can use the RepVGG network as the backbone network of the initial posture recognition model, the average pooling layer as the feature conversion layer of the initial posture recognition model, and the regression algorithm and the Euler angle prediction algorithm as the output layer of the initial posture recognition model to obtain the initial posture recognition model.

[0102] S202: Acquire a plurality of head image data and a rotation matrix corresponding to the plurality of head image data.

[0103] As a possible implementation method, after the recognition device constructs the initial posture recognition model, it can also construct the initial posture recognition model so as to obtain training data based on multiple head image data and rotation matrices corresponding to the multiple head image data to train the initial posture recognition model.

[0104] S203 , taking a plurality of head image data as samples and rotation matrices corresponding to the plurality of head image data as sample labels to obtain a plurality of sample data.

[0105] As a possible implementation method, after the recognition device obtains multiple head image data and rotation matrices corresponding to the multiple head image data, the multiple head image data can be used as samples and the rotation matrices corresponding to the multiple head image data as sample labels to obtain multiple sample data, thereby obtaining training data based on the multiple sample data.

[0106] For example, the recognition device may use the rotation matrix R of the plurality of head image data gt Convert to other forms and determine the rotation matrix R gt Rotation axis vector of logarithmic map on Lie algebra And the rotation matrix R gt Converted into Euler angle pyr_6d_gt, in radians, so that the loss function loss_6d_R of the regression algorithm and the loss function loss_6d_pyr of the Euler angle prediction algorithm are close in magnitude. gt Convert to Euler angle pyr_euler_gt, the unit is degree, and then encode pyr_euler_gt into integer part pyr_euler_gt_cls and remainder part pyr_euler_gt_res, the integer part is the difference between the current original label Euler angle and the minimum value and then modulo. Thus, the recognition device can use multiple head image data as samples, and the rotation matrix corresponding to the multiple head image data as sample labels to obtain multiple sample data.

[0107] The recognition device may determine the integer part and the remainder part of the Euler angle based on the following expression:

[0108] pyr_euler_gt_cls=(pyr_euler_gt-pyr_euler_gt_min)%m;

[0109] pyr_euler_gt_res=(pyr_euler_gt-m*pyr_euler_gt_cls) / m;

[0110] pyr_euler_gt=m*(pyr_euler_gt_cls+pyr_euler_gt_res);

[0111] C=(pyr_euler_gt_max-pyr_euler_gt_min)%m;

[0112] Among them, pyr_euler_gt_cls represents the integer part, pyr_euler_gt represents the Euler angle, pyr_euler_gt_min represents the minimum value of the Euler angle, m represents the modulus, pyr_euler_gt_res represents the remainder part, pyr_euler_gt_res∈[0,1), C represents the number of encoded integers, that is, the number of channels of the angle classification branch in the Euler angle prediction branch, and pyr_euler_gt_max represents the maximum value of the Euler angle. When m is 3, the encoding of 46 degrees is pyr_euler_gt_cls=15, pyr_euler_gt_res=1 / 3.

[0113] As an example, the recognition device transforms the rotation matrix R of the plurality of head image data into gt Convert Euler angles. Euler angles include pitch angle, yaw angle, and roll angle. Therefore, pitch angle, yaw angle, and roll angle all need to be encoded. The relationship between the sample label and label of a sample data is shown in Table 1:

[0114] Table 1

[0115]

[0116] As shown in Table 1, the rotation matrix R gt The rotation axis vector is obtained by logarithmic mapping and logarithmic mapping angle θ_gt, rotation matrix R gtConverted into Euler angle pyr_6d_gt and Euler angle pyr_euler_gt. The Euler angle pyr_6d_gt includes: pitch angle pyr_6d_gt_pitch, yaw angle pyr_6d_gt_yaw and roll angle pyr_6d_gt_roll. The Euler angle pyr_euler_gt includes: pitch angle pyr_euler_gt_pitch, yaw angle pyr_euler_gt_yaw and roll angle pyr_euler_gt_roll. The pitch angle pyr_euler_gt_pitch includes: integer part pyr_euler_gt_pitch_cls and remainder part pyr_euler_gt_pitch_res, the yaw angle pyr_euler_gt_yaw includes: integer part pyr_euler_gt_yaw_cls and remainder part pyr_euler_gt_yaw_res, and the roll angle pyr_euler_gt_roll includes: integer part pyr_euler_gt_roll_cls and remainder part pyr_euler_gt_roll_res.

[0117] S204: Perform image enhancement processing on multiple sample data to obtain training data.

[0118] As a possible implementation method, the recognition device takes multiple head image data as samples and the rotation matrices corresponding to the multiple head image data as sample labels. After obtaining the multiple sample data, the multiple sample data can be subjected to image enhancement processing to obtain training data, so as to train the initial posture recognition model based on the training data.

[0119] Exemplarily, the recognition device may perform image enhancement processing on multiple sample data in any one or more of the following ways: randomly adding rain, snow and / or fog noise in random proportions to multiple sample data, randomly adding noise and / or Gaussian noise to multiple sample data, randomly adding motion blur, Gaussian filter blur and / or mean filter blur to multiple sample data, randomly increasing or decreasing the contrast, saturation and / or brightness of a certain area of ​​multiple sample data, randomly swapping the red, green and blue (RGB) channels of multiple sample data, randomly converting multiple sample data into grayscale images, cropping multiple sample data up, down, left and right with a certain probability, flipping multiple sample data left and right with a certain probability, and making the same modifications, that is, changing the yaw angle and roll angle to the opposite of the original.

[0120] S205 . Train the initial posture recognition model based on the training data to obtain a posture recognition model.

[0121] As a possible implementation manner, after the recognition device performs image enhancement processing on a plurality of sample data to obtain training data, it can perform image enhancement processing on a plurality of sample data to obtain training data.

[0122] Exemplarily, the recognition device can determine the first output data of the regression algorithm and the second output data of the Euler angle prediction algorithm in the initial posture recognition model based on the training data, and determine the first loss value of the regression algorithm based on the first output data, and determine the second loss value of the Euler angle prediction algorithm based on the second output data. Thus, the recognition device can adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model to construct a posture recognition model.

[0123] In some other embodiments, the initial posture recognition model includes: a backbone network, a feature conversion layer and an output layer. The recognition device can also construct the initial posture recognition model by constructing the backbone network, the feature conversion layer and the output layer respectively. Figure 3 As shown, the above step S201, constructing an initial posture recognition model, can be specifically implemented as the following steps S2011-S2013:

[0124] S2011. Based on the RepVGG network, build the backbone network of the initial posture recognition model.

[0125] As a possible implementation method, the recognition device can construct a backbone network of the initial posture recognition model based on the RepVGG network.

[0126] Exemplarily, the recognition device can prune the RepVGG network, prune the number of VGG blocks in the RepVGG network from the original [2,4,14,1] to [2,4,8,1], and increase the number of channels of the last convolutional layer from 1280 to 1536. After pruning, the parameters of the initial posture recognition model are reduced and the amount of floating-point calculations is greatly reduced.

[0127] It should be understood that the recognition device uses a very small RepVGG network as the backbone network, which increases the number of parameters in the model feature mapping layer, but reduces the number of model parameters as a whole, reduces the model's demand for computing power, and also ensures the accuracy of the model. This network is smaller and more accurate than the 360DRepNet model. The 360DRepNet model uses RepVGG-B1g2 as a benchmark, with a parameter amount of 39.3M and a floating-point operation amount of 37GFlops. This application uses the smallest version of the RepVGG-S trimmed model as the backbone network, with a parameter amount of 7.87M and a floating-point operation amount of 1.39GFlops.

[0128] S2012. Based on the average pooling layer, a feature conversion layer of the initial posture recognition model is constructed.

[0129] As a possible implementation method, the recognition device can construct a feature conversion layer of the initial posture recognition model based on the average pooling layer.

[0130] Exemplarily, the recognition device can pass the output of the backbone network of the initial posture recognition model through an average pooling layer, convert the output into a tensor with a width and length of 1 (i.e., B*C*1*1), the number of channels C remains unchanged, and after a flattening operation, flatten the channel to convert the output into a tensor of B*C. Wherein, B is the number of samples for a small batch training, which is set according to the configuration of the training machine, and is not limited in the embodiment of the present application, and the value of C is 1536.

[0131] S2013. Based on the regression algorithm and Euler angle prediction algorithm, construct the output layer of the initial posture recognition model.

[0132] As a possible implementation method, the recognition device can construct an output layer of the initial posture recognition model based on a regression algorithm and an Euler angle prediction algorithm.

[0133] Exemplarily, the recognition device can input the output result of the feature conversion layer of the initial posture recognition model into the output layer of the initial posture recognition model, and the output layer of the initial posture recognition model is composed of two branches: a regression algorithm branch and an Euler angle prediction algorithm branch. The regression algorithm branch is a fully connected layer with 6 channels. The Euler angle prediction algorithm branch has 6 branches, of which 3 branches are branches for predicting the integer part categories of the 3 Euler angles (i.e., pitch angle pitch, yaw angle yaw, roll angle roll), the number of channels is the number of the integer part categories, and the other 3 branches are the parts of the 3 Euler angle remainders.

[0134] In some other embodiments, the recognition device can also determine the first output data of the regression algorithm and the second output data of the Euler angle prediction algorithm in the initial posture recognition model based on the training data, and determine the first loss value of the regression algorithm based on the first output data, and determine the second loss value of the Euler angle prediction algorithm based on the second output data. Thus, the recognition device can adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model to construct a posture recognition model. Therefore, if Figure 4 As shown, the above S205, training the initial posture recognition model based on the training data to obtain the posture recognition model, can be specifically implemented as the following steps S2051-S2055:

[0135] S2051. Based on the training data, respectively determine the first output data of the regression algorithm in the initial posture recognition model and the second output data of the Euler angle prediction algorithm.

[0136] As a possible implementation manner, the recognition device may determine the first output data of the regression algorithm and the second output data of the Euler angle prediction algorithm in the initial posture recognition model based on the training data.

[0137] For example, the recognition device can convert the output of the regression algorithm into the rotation matrix R pred , and convert the transformed rotation matrix into a logarithmic mapping Then the rotation matrix is ​​the Euler angle pyr_6d_pred, where the pitch angle is pyr_6d_pred_pitch, the yaw angle is pyr_6d_pred_yaw, and the roll angle is pyr_6d_pred_roll. The output of the Euler angle prediction algorithm has 6 sub-branches, of which 3 branches are the branches for the category prediction of the integer part of the 3 Euler angles, and the other 3 branches are the parts of the remainder of the 3 Euler angles. The recognition device can also process the results of these 6 branches through the sigmoid activation function to obtain the second output data through the following expression:

[0138] pyr_euler_pred=m*(pyr_euler_pred_cls+pyr_euler_pred_res);

[0139] Among them, pyr_euler_pred represents the predicted angle, m represents the modulus of the angle classification code, pyr_euler_pred_cls represents the index value with the largest angle classification probability (the index value with the largest value in the channel), and pyr_euler_pred_res represents the result of the regression branch after the sigmoid activation function, which is between 0 and 1.

[0140] S2052. Determine a first loss value of the regression algorithm based on the first output data.

[0141] As a possible implementation manner, the identification device may determine a first loss value of the regression algorithm based on the first output data.

[0142] Exemplarily, the recognition device may pred , pyr_6d_gt_pitch, pyr_6d_gt_yaw and pyr_6d_gt_roll are compared with the true label, and the first loss value is determined according to the loss calculation formula of the regression branch loss_6d=α1*loss_6d_R+β1*loss_6d_pyr. The recognition device can also determine the first loss value by the following expression:

[0143] loss_6d_pyr=Mse(pyr_6d_gt,pyr_6d_pred)=

[0144] (Mse(pyr_6d_gt_pitch,pyr_6d_pred_pitch)+Mse(pyr_6d_gt_yaw,pyr_6d_pred_yaw)+

[0145] Mse(pyr_6d_gt_roll,pyr_6d_pred_roll)) / 3;

[0146]

[0147] Among them, the loss function of the angle θ of the logarithmic mapping corresponding to the Lie algebra of the rotation matrix obtained by the regression branch, the rotation axis vector The loss function of the regression branch, the angle loss function of the two 3D vectors corresponding to the 6-dimensional output of the regression branch, and the Euler angle loss function corresponding to the rotation matrix are composed of four parts.

[0148] S2053. Determine a second loss value of the Euler angle prediction algorithm based on the second output data.

[0149] As a possible implementation manner, the identification device may determine a second loss value of the Euler angle prediction algorithm based on the second output data.

[0150] Exemplarily, the recognition device may determine the second loss value of the Euler angle prediction algorithm based on the second output data by using the following expression:

[0151]

[0152] Among them, the loss function of the Euler angle prediction branch is composed of the Euler angle classification (integer category branch) and regression branch (remainder branch) of the Euler angle prediction branch.

[0153] S2054: Adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model.

[0154] As a possible implementation method, after the recognition device determines the first loss value of the regression algorithm and the second loss value of the Euler angle prediction algorithm based on the first output data and the second output data, it can also adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model.

[0155] Exemplarily, the recognition device can use the sum of the first loss value and the second loss value as the total loss value, back-propagate the gradient loss, and use the SGD optimizer to optimize the initial posture recognition model. Because the overall loss value of the model is small, the learning rate is generally 10 -3 to 10 -5Iterate for a certain number of rounds (for example, 50-100 rounds) or stop iterating after the model converges.

[0156] Specifically, the two 3D column vectors corresponding to the 6D output tensor P of the regression algorithm are Take the first 3 dimensions on the channel of P as Take the last 3 dimensions on the channel of P as Let the rotation matrix be R, then The identification device can be expressed by the following expression The formula converted to R is:

[0157]

[0158]

[0159] Here, × represents the cross product of two vectors.

[0160] The recognition device can determine the total loss value of the initial posture recognition model training through the following expression:

[0161] Loss_total1=loss_6d+γ1*loss_euler_pyr=α1*loss_6d_R+β1*loss_6d_pyr+γ1*loss_euler_pyr=α1*(loss_6d_theta+loss_6d_a+loss_6d_omega)+β1*

[0162] loss_6d_pyr+γ1*loss_euler_pyr;

[0163]

[0164] loss_6d_pyr=Mse(pyr_6d_gt,pyr_6d_pred);

[0165] loss_euler_pyr=k*loss_euler_pyr_cls+l*loss_euler_pyr_res;

[0166] loss_euler_pyr_cls=CrossEntropyLoss(pyr_euler_gt_cls,pyr_euler_pred_cls);

[0167] loss_euler_pyr_res=Mse(pyr_euler_gt_res,pyr_euler_pred_res);

[0168] Among them, · represents the dot product of two vectors, Loss_total1 represents the total loss value of the initial posture recognition model training, α1, β1, γ1, k, l represent the weights of each loss, α1*loss_6d_R+β1*loss_6d_pyr represents the total loss of the regression algorithm, loss_6d_R represents the loss calculated by the rotation matrix, loss_6d_pyr represents the loss calculated by converting the rotation matrix of the regression branch prediction into the Euler angle method, loss_euler_pyr represents the loss of the Euler angle prediction algorithm, CrossEntropyLoss represents the cross entropy loss function, Mse represents the mean square error loss function, trace represents the trace of the matrix, arccos represents the inverse cosine function, R gt Represents the rotation matrix of the true label, R pred represents the predicted rotation matrix, Represents R pred The transpose of loss_6d_theta represents the rotation matrix R of the true label gt Rotate to the predicted rotation matrix R pred The required rotation matrix The loss of the angle θ corresponding to the logarithmic mapping of the Lie algebra characterizes R gt With R pred The geometric distance, loss_6d_a represents the rotation axis vector The loss represents the R gt With R pred The angle corresponding to the direction of the rotation axis, in radians, Representation vector Model, Representation vector The modulus of loss_6d_omega represents the two 3D column vectors corresponding to the 6-dimensional output tensor P. The angle between them is in radians. pyr_6d_gt represents the true label of the Euler angle, including pitch, yaw and roll angles, in radians. pyr_6d_pred represents the angle of the rotation matrix of the regression load converted to the Euler angle, including pitch, yaw and roll angles, in radians. loss_euler_pyr represents the loss of the Euler angle prediction algorithm, which consists of two losses: the classification (integer) loss loss_euler_pyr_cls of the Euler angle and the regression (remainder) loss loss_euler_pyr_res. There are 3 Euler angles (in degrees) and a total of 6 losses.

[0169] It should be understood that the Euler angle prediction algorithm is an auxiliary training branch. The purpose of introducing this algorithm is to allow the backbone network of the posture recognition model to learn as much image information as possible, to make the parameter distribution of the posture recognition model more even, to avoid the information of the posture recognition model being mainly concentrated in the subsequent fully connected layer, and to make the posture recognition model have smaller quantization loss when quantizing the posture recognition model, higher indicators after quantization, and better generalization ability of the posture recognition model.

[0170] S2055. Construct a posture recognition model based on the trained initial posture recognition model.

[0171] The output layer of the initial posture recognition model includes: a regression branch and an Euler angle prediction branch, and the output layer of the posture recognition model includes: a regression branch and / or an Euler angle prediction branch.

[0172] As a possible implementation manner, the recognition device may also construct a gesture recognition model based on the trained initial gesture recognition model.

[0173] Exemplarily, the recognition device adjusts the initial posture recognition model based on the first loss value and the second loss value, and after training the adjusted initial posture recognition model, the Euler angle prediction algorithm in the real posture recognition model can be deleted, and only the regression algorithm is retained. The recognition device re-parameterizes the backbone network of the initial posture recognition model, and uses a smaller learning rate (generally 10) to adjust the re-parameterized initial posture recognition model and parameters. -4 to 10 -6 Finally, the fine-tuned initial posture recognition model is quantized to obtain the posture recognition model.

[0174] Specifically, the recognition device can construct a posture recognition model based on the loss function of the regression branch of the following expression:

[0175] Loss_total2=α2*loss_6d_R+β2*loss_6d_pyr;

[0176] Among them, α2 and β2 represent the weights of the loss, which can be the same as or different from the above α1 and β1.

[0177] In some other embodiments, the recognition device may also use an SGD optimizer to iteratively optimize the initial posture recognition model based on the first loss value and the second loss value, and stop optimizing the initial posture recognition model when the number of iterations meets the first preset threshold or the initial posture recognition model converges, and train the optimized initial posture recognition model based on the training data. Figure 5As shown, the above step S2054, based on the first loss value and the second loss value, adjusts the initial posture recognition model, and trains the adjusted initial posture recognition model, can be specifically implemented as the following steps S2054a-S2054b:

[0178] S2054a. Based on the first loss value and the second loss value, use the SGD optimizer to iteratively optimize the initial posture recognition model.

[0179] As a possible implementation manner, the recognition device may use an SGD optimizer to iteratively optimize the initial posture recognition model based on the first loss value and the second loss value.

[0180] S2054b: When the number of iterations meets the first preset threshold or the initial posture recognition model converges, stop optimizing the initial posture recognition model, and train the optimized initial posture recognition model based on the training data.

[0181] As a possible implementation, when the number of iterations meets the first preset threshold or the initial posture recognition model converges, the recognition device can stop optimizing the initial posture recognition model and train the optimized initial posture recognition model based on the training data. When the number of iterations does not meet the first preset threshold or the initial posture recognition model does not converge, the recognition device can continue to iteratively optimize the initial posture recognition model using the SGD optimizer based on the first loss value and the second loss value.

[0182] In some other embodiments, the recognition device may also input the head image to be processed into the posture recognition model to obtain two three-dimensional vectors, convert the two three-dimensional vectors into a first rotation matrix and a second rotation matrix, and perform interpolation orthogonal processing on the first rotation matrix and the second rotation matrix to obtain a rotation matrix. Figure 6 As shown, the above step S102, inputting the head image to be processed into the posture recognition model to obtain a rotation matrix, can be specifically implemented as the following steps S1021-S1023:

[0183] S1021. Input the head image to be processed into a posture recognition model to obtain two three-dimensional vectors.

[0184] Among them, two three-dimensional vectors are used to represent the position information and direction information of the head in the head image to be processed.

[0185] As a possible implementation method, after the recognition device inputs the head image to be processed into the posture recognition model, the posture recognition model can output a 6-dimensional vector corresponding to the head image to be processed, and convert the 6-dimensional vector into two 3-dimensional vectors. The two 3-dimensional vectors represent the position and direction of the head in the head image to be processed in the coordinate system.

[0186] S1022. Based on the directions of the two three-dimensional vectors, the two three-dimensional column vectors are converted into a first rotation matrix and a second rotation matrix.

[0187] As a possible implementation method, the recognition device inputs the head image to be processed into the posture recognition model, and after obtaining two three-dimensional vectors, the two three-dimensional column vectors can be converted into a first rotation matrix and a second rotation matrix based on the directions of the two three-dimensional vectors.

[0188] For example, the posture recognition model is trained with only As a benchmark, it is directly assumed that and In the same direction, if the gesture recognition model is used in this way, there will be errors, so the recognition device needs to be based on (i.e., two three-dimensional vectors) to solve a rotation matrix (i.e., the first rotation matrix and the second rotation matrix). Therefore, the recognition device can be based on the following expressions respectively The direction determines the first rotation matrix R1 and based on The direction determines the second rotation matrix R2:

[0189]

[0190]

[0191] Here, × represents the cross product of two vectors.

[0192] S1023. Perform interpolation orthogonal processing on the first rotation matrix and the second rotation matrix to obtain a rotation matrix.

[0193] As a possible implementation method, after the recognition device converts two three-dimensional column vectors into a first rotation matrix and a second rotation matrix, it can interpolate the first rotation matrix and the second rotation matrix to obtain a first quaternion and a second quaternion, and then interpolate the first quaternion and the second quaternion to obtain a rotation matrix.

[0194] For example, the recognition device can interpolate the first rotation matrix R1 and the second rotation matrix R2. Since the interpolation is performed by taking the center point of R1 and R2, the quaternion linear interpolation method can be used for interpolation. The quaternion corresponding to the rotation matrices R1 and R2 is Then the recognition device can interpolate the rotation matrix through the following expression to obtain the rotation matrix:

[0195]

[0196] Among them, t is 0.5.

[0197] It should be understood that the interpolation process may select the midpoint of the first rotation matrix R1 and the second rotation matrix R2, or may select other positions of the first rotation matrix R1 and the second rotation matrix R2, and the embodiment of the present application does not limit this.

[0198] In some other embodiments, the recognition device may also determine the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, thereby determining whether the head posture in the head image to be processed is in an abnormal state based on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra. Figure 7 As shown, the head posture recognition method provided by the present application also includes the following steps S701-S702:

[0199] S701. Determine the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0200] As a possible implementation manner, the recognition device may also rotate the rotation axis vector of the logarithmic mapping of the matrix on the Lie algebra, so as to determine whether the head posture in the head image to be processed is in an abnormal state based on the rotation axis vector.

[0201] Exemplarily, the identification device can determine the angle of the logarithmic mapping of the rotation matrix on the Lie algebra, and determine the eigenvector corresponding to the eigenvalue of the rotation matrix when it is 1 based on the angle of the logarithmic mapping, thereby determining the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0202] S702: Determine whether the head posture in the head image to be processed is in an abnormal state based on at least the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0203] As a possible implementation method, after the recognition device determines the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, it can determine whether the head posture in the head image to be processed is in an abnormal state based on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0204] Exemplarily, the recognition device may compare the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra with the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra corresponding to the normal posture, or compare the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra with the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra corresponding to the abnormal posture, to determine whether the head posture in the head image to be processed is in an abnormal state.

[0205] In some other embodiments, the recognition device may also determine the logarithmic mapping angle of the rotation matrix on the Lie algebra, thereby determining whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector. Figure 8 As shown, the head posture recognition method provided by the present application also includes the following steps S700:

[0206] S700, determining the logarithmic mapping angle of the rotation matrix on the Lie algebra.

[0207] As a possible implementation method, the recognition device can also determine the logarithmic mapping angle of the rotation matrix on the Lie algebra, and thus determine whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector.

[0208] Exemplarily, the recognition device may determine the logarithmic mapping angle of the rotation matrix on the Lie algebra by the following expression:

[0209]

[0210] Among them, θ represents the logarithmic mapping angle of the rotation matrix on the Lie algebra, The rotation axis vector representing the logarithmic mapping of the rotation matrix on the Lie algebra.

[0211] It should be understood that the execution order of the above-mentioned step S700 and step S701 is not specific, or step S700 and step S701 can be determined at the same time, and the embodiment of the present application is not limited to this.

[0212] It should be understood that after the recognition device determines the logarithmic mapping angle of the rotation matrix on the Lie algebra, it can determine whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector. Therefore, the above step S702, at least based on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, determines whether the head posture in the head image to be processed is in an abnormal state, which can be specifically implemented as: based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector, determines whether the head posture in the head image to be processed is in an abnormal state.

[0213] Exemplarily, the recognition device can determine the target logarithmic mapping angle and target rotation axis vector for rotating from a standard normal posture to the head posture in the head image to be processed, thereby determining that the head posture of the head image to be processed is an abnormal posture when at least one of the following conditions is met: the target logarithmic mapping angle is within the first angle range, the target logarithmic mapping angle is outside the second angle range, the target rotation axis vector is within the first vector range, and the target rotation axis vector is outside the second vector range.

[0214] As another example, the recognition device may determine that the head posture of the head image to be processed is an abnormal posture based on comparing the logarithmic mapping angle and rotation axis vector of the rotation matrix of the head image to be processed with the logarithmic mapping angle and rotation axis vector of a standard normal posture.

[0215] In some other embodiments, the recognition device may also determine whether the head posture in the head image to be processed is an abnormal posture by determining the target logarithmic mapping angle and the target rotation axis vector from the standard normal posture to the head posture in the head image to be processed. Fig. 9 As shown, the above-mentioned determination of whether the head posture in the head image to be processed is in an abnormal state based on the angle and the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra can be specifically implemented as the following steps S901-S902:

[0216] S901. Determine a target logarithmic mapping angle and a target rotation axis vector for rotating from a standard normal posture to a head posture in a head image to be processed based on the logarithmic mapping angle and the rotation axis vector of the rotation matrix on Lie algebra.

[0217] As a possible implementation method, the recognition device can determine the target logarithmic mapping angle and target rotation axis vector of the head posture rotated from the standard normal posture to the head posture in the head image to be processed based on the rotation matrix of the head image to be processed and the rotation matrix of the standard normal posture.

[0218] Exemplarily, the recognition device may determine the target logarithmic mapping angle and the target rotation axis vector by the following expression:

[0219]

[0220] Among them, θ represents the target logarithmic mapping angle, Represents the target rotation axis vector.

[0221] S902. Determine that the head posture of the head image to be processed is an abnormal posture when at least one of the following conditions is met: the target logarithmic mapping angle is within a first angle range, the target logarithmic mapping angle is outside a second angle range, the target rotation axis vector is within a first vector range, and the target rotation axis vector is outside a second vector range.

[0222] As a possible implementation, when at least one of the following conditions is met, the recognition device can determine that the head posture of the head image to be processed is an abnormal posture: the target logarithmic mapping angle is within the first angle range, the target logarithmic mapping angle is outside the second angle range, the target rotation axis vector is within the first vector range, and the target rotation axis vector is outside the second vector range. When the above conditions are not met, it means that the head in the head image to be processed is within the range of normal posture. Among them, the first angle range and the first vector range represent the range corresponding to the logarithmic mapping angle and the rotation axis vector of the rotation matrix of the head in a normal posture, the second angle range and the second vector range represent the range corresponding to the logarithmic mapping angle and the rotation axis vector of the rotation matrix of the head in an abnormal posture, and the specific values ​​of the first angle range, the second angle range, the first vector, and the second vector range are determined by relevant management personnel according to actual conditions, and the embodiments of the present application do not limit this.

[0223] In some other embodiments, the recognition device can also determine whether the head posture in the head image to be processed is an abnormal posture by using the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector. Therefore, the above determination of whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector can be specifically implemented as determining that the head posture of the head image to be processed is an abnormal posture when the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold.

[0224] As a possible implementation, when the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold, the recognition device can determine that the head posture of the head image to be processed is an abnormal posture. When the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture meets the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture meets the third preset threshold, it means that the head in the head image to be processed is within the range of normal posture. Among them, the second preset threshold and the third preset threshold represent the maximum difference between the logarithmic mapping angle and the rotation axis vector of the rotation matrix on the Lie algebra and the logarithmic mapping angle and the rotation axis vector of the standard normal posture when the head in the head image to be processed is in the normal posture range. The specific values ​​of the second preset threshold and the third preset threshold are determined by relevant management personnel according to actual conditions, and the embodiments of the present application do not limit this.

[0225] It should be understood that the recognition device can not only judge whether the head posture of the head image to be processed is an abnormal posture by the maximum difference between the logarithmic mapping angle and rotation axis vector of the rotation matrix on the Lie algebra and the logarithmic mapping angle and rotation axis vector of the standard normal posture, but also judge whether the head posture of the head image to be processed is an abnormal posture by the maximum difference between the logarithmic mapping angle and rotation axis vector of the rotation matrix on the Lie algebra and the logarithmic mapping angle and rotation axis vector of the abnormal posture. The embodiment of the present application does not limit the specific method of judging whether the head posture of the head image to be processed is an abnormal posture.

[0226] In some other embodiments, the head image to be processed is the head image of the vehicle driver, or the head image of a pedestrian within the image acquisition range of the vehicle. The recognition device can also determine whether the vehicle driver or the pedestrian within the image acquisition range of the vehicle has abnormal behavior based on the rotation matrix of the head image to be processed, and issue a warning message if abnormal behavior exists. Fig.10 As shown, the head posture recognition method provided by the present application also includes the following steps S1001-S1002:

[0227] S1001. Determine a ratio of head images to be processed with abnormal postures among a plurality of head images to be processed within a preset period.

[0228] As a possible implementation method, after the recognition device determines whether the head posture in the head image to be processed is an abnormal posture, it can determine the ratio of the head images to be processed with abnormal postures in multiple head images to be processed within a preset period, so as to determine whether the vehicle driver or the pedestrian within the image acquisition range of the vehicle has abnormal behavior based on the ratio. Among them, the preset period is set by relevant management personnel based on actual conditions, and the embodiments of the present application do not limit this.

[0229] S1002: When the ratio is greater than a fourth preset threshold, issue an alarm message.

[0230] Among them, the warning information is used to notify the vehicle's automatic driving system to take over the vehicle and issue an alarm message for dangerous driving.

[0231] As a possible implementation manner, after the recognition device determines the ratio of the head images to be processed with abnormal postures among the multiple head images to be processed within a preset period, the recognition device can issue an alarm message when the ratio is greater than a fourth preset threshold.

[0232] Exemplarily, when the ratio is greater than the fourth preset threshold, it means that the head posture of the vehicle driver or the pedestrian within the image acquisition range of the vehicle is in an abnormal posture for a long time, and there is a risk of dangerous driving. The recognition device can issue a warning message to notify the vehicle's automatic driving system to take over the vehicle to avoid an accident. When the ratio is less than the fourth preset threshold, it means that the head posture of the vehicle driver or the pedestrian within the image acquisition range of the vehicle is only in an abnormal posture occasionally, which does not affect the safe driving of the vehicle.

[0233] It should be understood that if the driver of the vehicle does not look straight ahead for a long time, but looks to the left rear area, and the head posture mostly falls in the abnormal area, and the abnormal state ratio exceeds the fourth preset threshold, it is determined that the driver is distracted driving, and the recognition device can issue a warning message to notify the vehicle's automatic driving system to take over the vehicle. When young pedestrians (such as children) on the road suddenly turn their heads to the rear when crossing the road, the recognition device can issue a warning message, and the automatic driving system will take over and slow down or stop the vehicle. After the pedestrian's state stabilizes or the pedestrian moves to a safe area, it continues to drive to prevent pedestrians from running or running around on the road and causing accidents.

[0234] Fig.11 is a diagram showing a gesture recognition model architecture according to an exemplary embodiment. Fig.11As shown, multiple head image data images([B,,,]) are obtained, and image enhancement processing image_tensor([B,3,224,224]) is performed on the multiple head image data. The multiple head image data after enhancement processing are input into the backbone network of the initial posture recognition model, backbone([B,1536,7,7]), and the output of the backbone network is input into the average pooling layer AdaptiveAvgPool2d([B,1536,1,1]) of the initial posture recognition model. The output of the average pooling layer Perform multi-dimensional tensor flatten([B,1536]), and train flatten([B,1536]) through the auxiliary training branch - Euler angle prediction branch. The output of flatten([B,1536]) is two three-dimensional vectors fc_6d_reg([B,6]), which is converted into a rotation matrix R_6d([B,3,3]). The rotation matrix R_6d([B,3,3]) includes: 6d_a([B,3]) and pyr_6d([B,3]). Among them, the auxiliary training branch - Euler angle prediction branch includes: yaw submodule, pitch submodule and roll submodule. The yaw submodule includes yaw_cls([B,120]), yaw_bias_reg([B,1]) and yaw([B,1]). The pitch submodule includes: pitch_cls([B,66]), pitch_bias_reg([B,1]), and pitch([B,1]). The roll submodule includes: roll_cls([B,66]), roll_bias_reg([B,1]), and roll([B,1]).

[0235] Fig.12 is a schematic diagram of abnormal posture recognition according to an exemplary embodiment. Fig.12 As shown, the middle dotted line in the figure represents the center of the reference posture, the angle represents the maximum range of the reference posture, theta_max is the maximum angle, theta_current is the current angle, the reference posture represents the normal posture, and the figure shows that the angle of the current posture falls within the maximum range of the reference posture, which means that the head in the head image corresponding to the current posture is in a normal posture.

[0236] Fig.13 is a schematic diagram of a gesture recognition model structure according to an exemplary embodiment. Fig.13As shown, the posture recognition model includes: backbone network, average pooling layer, wh is changed to 1, tensor is flattened in channel dimension, 6D regression branch FC (channel 6), 6D regression branch includes: 6D degrees of freedom, 2 3D vectors are converted into rotation matrix, orthogonal method conversion, 6D regression loss function and rotation matrix are converted into Euler angle, 6D regression loss function includes: loss function of logarithmic mapping of rotation matrix, and conversion of rotation matrix into Euler angle includes: 6D regression Euler angle loss function. Euler angle prediction branch: yaw angle, pitch angle, roll angle, Euler angle loss function, initial training is performed using logarithmic mapping loss function and Euler angle loss function, fine-tuning training is performed using logarithmic mapping loss function and 6D regression Euler angle loss function, and during model inference, 6D is converted into rotation matrix, and posture comparison is performed using logarithmic mapping corresponding to the rotation matrix.

[0237] The posture recognition model uses a very small RepVGG network as the backbone network, which slightly increases the number of parameters in the model feature mapping layer, but overall reduces the number of model parameters and the model's demand for computing power, while also ensuring the accuracy of the model. The network is smaller than the 6DRepNet model and has high accuracy. The RepVGG-B1g2 model used as the benchmark has 39.3M parameters and 37GFlops of floating-point operations, while the pruned RepVGG-S model of the posture recognition model uses 7.87M parameters and 1.39GFlops of floating-point operations as the backbone network. The average error of the posture recognition model on the AFLW2000 data is 2.17 degrees.

[0238] Fig.14 is a flowchart of another head posture recognition method according to an exemplary embodiment. Fig.14 As shown, the method includes the following steps: S1401, initial model training (equivalent to the above step S205, training the initial posture recognition model based on the training data to obtain the posture recognition model). S1402, removing the Euler angle prediction branch, re-parameterizing the model, and re-parameterizing the model parameters of the initial training. S1403, model fine-tuning training (equivalent to the above step S2055, constructing a posture recognition model based on the trained initial posture recognition model). S1404, model quantization and reasoning.

[0239] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the head posture recognition device or electronic device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0240] The embodiment of the present application can exemplarily divide the functional modules of the head posture recognition device or electronic device according to the above method. For example, the head posture recognition device or electronic device can include various functional modules corresponding to the various functional divisions, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0241] Fig.15 is a block diagram of a head posture recognition device according to an exemplary embodiment. Fig.15 The head posture recognition device 1500 includes: an acquisition module 1501 and a processing module 1502; the acquisition module 1501 is used to acquire a head image to be processed; the processing module 1502 is used to input the head image to be processed into a posture recognition model to obtain a rotation matrix; the posture recognition model is trained based on the loss function of the regression algorithm; the loss function of the regression algorithm includes a loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra, which is used to reflect the angle between the predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and the rotation axis vector of the logarithmic mapping of the real rotation matrix on the corresponding Lie algebra; the rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

[0242] In a possible implementation, the regression algorithm is a 6D regression algorithm, and the loss function of the regression algorithm also includes: a loss function of the logarithmic mapping angle of the rotation matrix on the Lie algebra, a loss function of the angle between two three-dimensional vectors output by the posture recognition model, and a mean square error loss function.

[0243] In another possible implementation, the posture recognition model is also trained based on the loss function of the Euler angle prediction algorithm; the loss function of the Euler angle prediction algorithm includes: a loss function of the integer part of the angle of the Euler angle and a loss function of the remainder part of the angle of the Euler angle. In yet another possible implementation, the above-mentioned device also includes: a construction module 1503 and a training module 1504. The construction module 1503 is used to construct an initial posture recognition model; the acquisition module 1501 is also used to obtain multiple head image data and rotation matrices corresponding to the multiple head image data; the processing module 1502 is also used to use the multiple head image data as samples and the rotation matrices corresponding to the multiple head image data as sample labels to obtain multiple sample data; the processing module 1502 is also used to perform image enhancement processing on the multiple sample data to obtain training data; the training module 1504 is used to train the initial posture recognition model based on the training data to obtain a posture recognition model.

[0244] In another possible implementation, the initial posture recognition model includes: a backbone network, a feature conversion layer and an output layer; a construction module 1503 is specifically used to construct the backbone network of the initial posture recognition model based on the RepVGG network; based on the average pooling layer, the feature conversion layer of the initial posture recognition model is constructed; based on the regression algorithm and the Euler angle prediction algorithm, the output layer of the initial posture recognition model is constructed.

[0245] In another possible implementation, the training module 1504 is specifically used to determine, based on the training data, the first output data of the regression algorithm in the initial posture recognition model and the second output data of the Euler angle prediction algorithm; determine the first loss value of the regression algorithm based on the first output data; determine the second loss value of the Euler angle prediction algorithm based on the second output data; adjust the initial posture recognition model based on the first loss value and the second loss value, and train the adjusted initial posture recognition model; construct a posture recognition model based on the trained initial posture recognition model; the output layer of the initial posture recognition model includes: a regression branch and a Euler angle prediction branch; the output layer of the posture recognition model includes: a regression branch and / or a Euler angle prediction branch.

[0246] In another possible implementation, the training module 1504 is specifically used to iteratively optimize the initial posture recognition model using an SGD optimizer based on the first loss value and the second loss value; when the number of iterations meets the first preset threshold or the initial posture recognition model converges, the optimization of the initial posture recognition model is stopped, and the optimized initial posture recognition model is trained based on the training data.

[0247] In another possible implementation, the processing module 1502 is specifically used to input the head image to be processed into a posture recognition model to obtain two three-dimensional vectors; the two three-dimensional vectors are used to represent the position information and direction information of the head in the head image to be processed; based on the directions of the two three-dimensional vectors, the two three-dimensional column vectors are converted into a first rotation matrix and a second rotation matrix; and the first rotation matrix and the second rotation matrix are interpolated and orthogonally processed to obtain the rotation matrix.

[0248] In another possible implementation, the apparatus further includes: a determination module 1505. The determination module 1505 is used to determine the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra; the determination module 1505 is further used to determine whether the head posture in the head image to be processed is in an abnormal state based on at least the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

[0249] In another possible implementation, the determination module 1505 is also used to determine the logarithmic mapping angle of the rotation matrix on the Lie algebra; the determination module 1505 is specifically used to determine whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector.

[0250] In another possible implementation, the determination module 1505 is specifically used to determine the target logarithmic mapping angle and the target rotation axis vector required to rotate from the standard normal posture to the head posture in the head image to be processed based on the logarithmic mapping angle of the rotation matrix on the Lie algebra and the rotation axis vector; when at least one of the following conditions is met, the head posture of the head image to be processed is determined to be an abnormal posture: the target logarithmic mapping angle is within the first angle range, the target logarithmic mapping angle is outside the second angle range, the target rotation axis vector is within the first vector range, and the target rotation axis vector is outside the second vector range.

[0251] In another possible implementation, the determination module 1505 is specifically used to determine that the head posture of the head image to be processed is an abnormal posture when the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold.

[0252] In another possible implementation, the head image to be processed is a head image of a vehicle driver, or a head image of a pedestrian within the image acquisition range of the vehicle; the above device further includes: an alarm module 1506. The determination module 1505 is further used to determine the ratio of the head images to be processed with abnormal postures in a plurality of head images to be processed within a preset period; the alarm module 1506 is used to issue an alarm message when the ratio is greater than a fourth preset threshold; the alarm message is used to notify the vehicle's automatic driving system to take over the vehicle and issue an alarm message of dangerous driving.

[0253] According to the above technical means, the regression algorithm can provide an accurate three-dimensional rotation matrix. On the basis of the loss function of the regression algorithm, the loss function of the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra is added, which avoids the oscillation of the loss function of the regression algorithm in the training process in the related technology, reduces the error of the rotation matrix output by the posture recognition model, and improves the accuracy of the rotation matrix, so that the rotation matrix can more accurately represent the state of the head in the head image to be processed, so that the posture recognition model can more accurately identify the position, direction and rotation angle of the head in the space in the head image to be processed.

[0254] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0255] Fig.16 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Fig.16 As shown, the electronic device 1600 includes but is not limited to: a processor 1601 and a memory 1602 .

[0256] The memory 1602 is used to store executable instructions of the processor 1601. It can be understood that the processor 1601 is configured to execute instructions to implement the head posture recognition method in the above embodiment.

[0257] It should be noted that those skilled in the art can understand that Fig.16 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include Fig.16 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0258] The processor 1601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1602, and calling data stored in the memory 1602, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 1601 may include one or more processing units. Optionally, the processor 1601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1601.

[0259] The memory 1602 may be used to store software programs and various data. The memory 1602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one functional module (such as a determination unit, a processing unit, etc.), etc. In addition, the memory 1602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0260] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1602 including instructions, and the above instructions can be executed by the processor 1601 of the electronic device 1600 to implement the head posture recognition method in the above embodiment.

[0261] In actual implementation, Fig.15 The functions of the acquisition module 1501, the processing module 1502, the construction module 1503, the training module 1504, the determination module 1505 and the alarm module 1506 in the embodiment can be realized by Fig.16 The processor 1601 in the embodiment calls the computer program stored in the memory 1602 to implement. The specific execution process can refer to the description of the method part in the above embodiment, which will not be repeated here.

[0262] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0263] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, which can be executed by the processor 1601 of the electronic device to complete the head posture recognition method in the above embodiment.

[0264] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the electronic device, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect as the above-mentioned method can be achieved. To avoid repetition, they will not be repeated here.

[0265] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0266] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0267] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0268] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0269] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD and other media that can store program code.

[0270] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A head posture recognition method, characterized in that: The method comprises: Obtaining a head image to be processed; The head image to be processed is input into a posture recognition model to obtain a rotation matrix; the posture recognition model is trained based on a loss function of a regression algorithm; the loss function of the regression algorithm includes a loss function of a rotation axis vector of a logarithmic mapping of the rotation matrix on a Lie algebra, which is used to reflect the angle between a predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and a rotation axis vector of a logarithmic mapping of a true rotation matrix on a corresponding Lie algebra; the rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

2. The method according to claim 1, characterized in that The regression algorithm is a 6D regression algorithm; the loss function of the regression algorithm also includes: a loss function of the logarithmic mapping angle of the rotation matrix on Lie algebra, a loss function of the angle between two three-dimensional vectors output by the posture recognition model, and a mean square error loss function.

3. The method according to claim 1, characterized in that The posture recognition model is also trained based on the loss function of the Euler angle prediction algorithm; The loss function of the Euler angle prediction algorithm includes: a loss function of the angular integer part of the Euler angle and a loss function of the angular remainder part of the Euler angle.

4. The method according to claim 1, characterized in that: The method further comprises: Construct an initial posture recognition model; Acquire a plurality of head image data and a rotation matrix corresponding to the plurality of head image data; Taking the multiple head image data as samples and the rotation matrices corresponding to the multiple head image data as sample labels, a plurality of sample data are obtained; Performing image enhancement processing on the plurality of sample data to obtain training data; The initial gesture recognition model is trained based on the training data to obtain the gesture recognition model.

5. The method according to claim 4, characterized in that The initial posture recognition model includes: a backbone network, a feature conversion layer and an output layer; the construction of the initial posture recognition model includes: Based on the RepVGG network, construct the backbone network of the initial posture recognition model; Based on the average pooling layer, construct a feature conversion layer of the initial posture recognition model; Based on the regression algorithm and the Euler angle prediction algorithm, an output layer of the initial posture recognition model is constructed.

6. The method according to claim 4, characterized in that The step of training the initial posture recognition model based on the training data to obtain the posture recognition model includes: Based on the training data, respectively determine the first output data of the regression algorithm and the second output data of the Euler angle prediction algorithm in the initial posture recognition model; determining a first loss value of the regression algorithm based on the first output data; Determine a second loss value of the Euler angle prediction algorithm based on the second output data; Based on the first loss value and the second loss value, adjusting the initial posture recognition model, and training the adjusted initial posture recognition model; The posture recognition model is constructed based on the trained initial posture recognition model; the output layer of the initial posture recognition model includes: a regression branch and an Euler angle prediction branch; the output layer of the posture recognition model includes: the regression branch and / or the Euler angle prediction branch.

7. The method according to claim 6, characterized in that The adjusting the initial posture recognition model based on the first loss value and the second loss value, and training the adjusted initial posture recognition model, comprises: Based on the first loss value and the second loss value, using an SGD optimizer to iteratively optimize the initial posture recognition model; When the number of iterations meets a first preset threshold or the initial gesture recognition model converges, the optimization of the initial gesture recognition model is stopped, and the optimized initial gesture recognition model is trained based on the training data.

8. The method according to claim 1, characterized in that: The step of inputting the head image to be processed into a posture recognition model to obtain a rotation matrix includes: Inputting the head image to be processed into the posture recognition model to obtain two three-dimensional vectors; the two three-dimensional vectors are used to represent the position information and direction information of the head in the head image to be processed; Based on the directions of the two three-dimensional vectors, the two three-dimensional column vectors are converted into a first rotation matrix and a second rotation matrix; Performing interpolation orthogonal processing on the first rotation matrix and the second rotation matrix to obtain the rotation matrix.

9. The method according to claim 1, characterized in that: The method further comprises: Determine the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra; Whether the head posture in the head image to be processed is in an abnormal state is determined based at least on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra.

10. The method according to claim 9, characterized in that The method further comprises: Determining a logarithmic mapping angle of the rotation matrix on the Lie algebra; The step of determining whether the head posture in the head image to be processed is in an abnormal state based at least on the rotation axis vector of the logarithmic mapping of the rotation matrix on the Lie algebra comprises: Based on the angle of the logarithmic mapping of the rotation matrix on the Lie algebra and the rotation axis vector, it is determined whether the head posture in the head image to be processed is in an abnormal state.

11. The method according to claim 10, characterized in that The step of determining whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on Lie algebra and the rotation axis vector comprises: Determine a target logarithmic mapping angle and a target rotation axis vector required for rotating from a standard normal posture to a head posture in the head image to be processed based on the logarithmic mapping angle of the rotation matrix on Lie algebra and the rotation axis vector; When at least one of the following conditions is met, it is determined that the head posture of the head image to be processed is an abnormal posture: The target logarithmic mapping angle is within a first angle range, the target logarithmic mapping angle is outside a second angle range, the target rotation axis vector is within a first vector range, and the target rotation axis vector is outside a second vector range.

12. The method according to claim 10, characterized in that The step of determining whether the head posture in the head image to be processed is in an abnormal state based on the logarithmic mapping angle of the rotation matrix on Lie algebra and the rotation axis vector comprises: When the difference between the logarithmic mapping angle of the rotation matrix on the Lie algebra and the logarithmic mapping angle of the standard normal posture does not meet the second preset threshold, or the difference between the rotation axis vector and the rotation axis vector of the standard normal posture does not meet the third preset threshold, it is determined that the head posture of the head image to be processed is an abnormal posture.

13. The method according to claim 9, characterized in that The head image to be processed is a head image of a vehicle driver, or a head image of a pedestrian within the image acquisition range of the vehicle; the method further includes: Determine a ratio of head images to be processed with abnormal postures among a plurality of head images to be processed within a preset period; When the ratio is greater than a fourth preset threshold, a warning message is issued; the warning message is used to notify the automatic driving system of the vehicle to take over the vehicle and issue a warning message of dangerous driving.

14. A head posture recognition device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire the head image to be processed; The processing module is used to input the head image to be processed into a posture recognition model to obtain a rotation matrix; the posture recognition model is trained based on a loss function of a regression algorithm; the loss function of the regression algorithm includes a loss function of a rotation axis vector of a logarithmic mapping of the rotation matrix on a Lie algebra, which is used to reflect the angle between a predicted rotation matrix obtained by training the posture recognition model based on the regression algorithm and a rotation axis vector of a logarithmic mapping of a true rotation matrix on a corresponding Lie algebra; the rotation matrix is ​​used to characterize the posture of the head in the head image to be processed in three-dimensional space.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is capable of performing the method as claimed in any one of claims 1 to 13.

Images