Human head posture recognition, model building method, device, equipment and storage medium
By constructing a head pose recognition model, using the direct mapping relationship between two-dimensional images and the rotation matrix, the universal joint deadlock problem when Euler angles representing head poses is solved, and higher accuracy and efficiency recognition is achieved.
Patent Information
- Application Number
- CN202010074231.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-01-22
- Publication Date
- 2025-08-08
- Estimated Expiration
- 2040-01-22
AI Technical Summary
In the prior art, the recognition of human head postures has a problem of universal joint deadlock when expressing human head postures using Euler angles, resulting in low recognition accuracy and large discontinuity.
The head pose recognition model is adopted. By inputting two-dimensional images into the trained neural network model, the rotation matrix is used to represent the rotation angle of the head features. The model is trained with the two-dimensional image as the sample and the rotation matrix as the label to build a direct mapping relationship.
It improves the accuracy and recognition efficiency of head posture recognition, avoids the discontinuity problem caused by universal joint deadlock, and has stronger semantic features.
Smart Images
Figure CN113158708B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for human head posture recognition and model building. Background Art
[0002] Head rotation angle recognition, also known as head posture recognition, is used to identify the direction of a person's face. This technology can be applied in surveillance applications. For example, it can monitor a driver's attentiveness during long-distance driving. By monitoring the driver's facial movements, accidents caused by inattention, such as drowsiness, can be avoided.
[0003] In related technologies, the orientation of a person's head posture is expressed using Euler angles, and the head posture is obtained by regressing the Euler angles. However, the Euler angles themselves have the problem of gimbal deadlock. Thus, the discontinuity caused by the gimbal deadlock can lead to large differences between the Euler angles regressed from head images of similar angles, thereby reducing the recognition accuracy. Summary of the Invention
[0004] To overcome the problems existing in the related art, the present application provides a method, device, equipment and storage medium for head posture recognition and model construction.
[0005] According to a first aspect of an embodiment of the present application, a method for recognizing a head posture is provided, the method comprising:
[0006] Acquire a two-dimensional image;
[0007] The two-dimensional image is input into a head posture recognition model, the head features in the two-dimensional image are identified, and a rotation matrix of the head features is output, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image including at least a head as a sample and the rotation matrix of the sample as a label.
[0008] According to a second aspect of an embodiment of the present application, a method for constructing a head posture recognition model is provided, the method comprising:
[0009] Acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0010] The two-dimensional image is used as a sample and the rotation matrix of the sample is used as a label to train the set neural network model to construct the head posture recognition model.
[0011] According to a third aspect of an embodiment of the present application, a head posture recognition device is provided, the device comprising:
[0012] an acquisition module configured to acquire a two-dimensional image;
[0013] The output module is configured to input the two-dimensional image into a head posture recognition model, identify the head features in the two-dimensional image, and output the rotation matrix of the head features, wherein the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image of at least a head as a sample and the rotation matrix of the sample as a label.
[0014] According to a fourth aspect of an embodiment of the present application, a device for constructing a head posture recognition model is provided, the device comprising:
[0015] an acquisition module configured to acquire a plurality of two-dimensional images at least including a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0016] The construction module is configured to train the set neural network model using the two-dimensional image as a sample and the rotation matrix of the sample as a label to construct the head posture recognition model.
[0017] According to a fifth aspect of an embodiment of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:
[0018] Acquire a two-dimensional image;
[0019] The two-dimensional image is input into a head posture recognition model, the head features in the two-dimensional image are identified, and a rotation matrix of the head features is output, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image including at least a head as a sample and the rotation matrix of the sample as a label.
[0020] According to a sixth aspect of an embodiment of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:
[0021] Acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0022] The two-dimensional image is used as a sample and the rotation matrix of the sample is used as a label to train the set neural network model to construct the head posture recognition model.
[0023] According to a seventh aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the following steps are implemented:
[0024] Acquire a two-dimensional image;
[0025] The two-dimensional image is input into a head posture recognition model, the head features in the two-dimensional image are identified, and a rotation matrix of the head features is output, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image including at least a head as a sample and the rotation matrix of the sample as a label.
[0026] According to an eighth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the following steps are implemented:
[0027] Acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0028] The two-dimensional image is used as a sample and the rotation matrix of the sample is used as a label to train the set neural network model to construct the head posture recognition model.
[0029] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:
[0030] The head posture recognition method in the present technical solution inputs a two-dimensional image including a head into a trained head posture recognition model. After recognition by the head posture recognition model, the rotation matrix corresponding to the head features in the two-dimensional image can be directly obtained, and the rotation matrix is used to represent the rotation angle of the head features. Compared with the related art that uses Euler angles to represent the rotation angle of the head features, the present technical solution can avoid the problem of large differences between the Euler angles regressed from head features of similar angles due to discontinuity caused by universal joint deadlock, can effectively improve the recognition accuracy, and has more semantic features; and the head posture recognition model in the present technical solution is constructed by training using multiple two-dimensional images including heads as samples and the rotation matrices of the samples as labels, which can also further improve the recognition accuracy; using the head posture recognition model to construct a direct mapping relationship between the two-dimensional image and the rotation matrix, the rotation angle of the head features can be quickly obtained, thereby improving the recognition efficiency.
[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0033] Figure 1The figure is a flowchart of a method for head posture recognition shown as an exemplary embodiment of the present application.
[0034] Figure 2 The figure is a flowchart of a head posture recognition model recognition process shown as an exemplary embodiment of the present application.
[0035] Figure 3 This is a flowchart of the recognition process of a two-dimensional image input into a head posture recognition model as shown in an exemplary embodiment of the present application.
[0036] Figure 4 This is a flow chart of a method for constructing a head posture recognition model shown in an exemplary embodiment of the present application.
[0037] Figure 5 This is a flow chart of a method for constructing a head posture recognition model shown in an exemplary embodiment of the present application.
[0038] Figure 6A The figure is a flowchart of a process for obtaining a rotation matrix corresponding to a two-dimensional image, shown in an exemplary embodiment of the present application.
[0039] Figure 6B This is a schematic diagram of constructing a three-dimensional model using a two-dimensional image, as shown in an exemplary embodiment of the present application.
[0040] Figure 7 The figure is a schematic structural diagram of a head posture recognition device according to an exemplary embodiment of the present application.
[0041] Figure 8 This is a schematic structural diagram of a device for constructing a head posture recognition model according to an exemplary embodiment of the present application.
[0042] Figure 9 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment of the present application.
[0043] Figure 10 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0046] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0047] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0048] The following is a detailed description of the head posture recognition, model building method, device, equipment and storage medium of the present application in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.
[0049] This application provides a head posture recognition method applicable to scenarios such as traffic monitoring. The method can be executed by a monitoring device, such as one installed in a vehicle. By executing the head posture recognition method on the monitoring device, the head posture of a driver driving the vehicle can be determined, thereby determining whether the driver is focused on driving. This head posture recognition method can also be applied to other application scenarios, which are not specifically limited by this application. Figure 1 FIG. 1 is a flow chart of a method for recognizing a head posture according to an exemplary embodiment of the present application. Figure 1 As shown, the head posture recognition method includes the following steps 101 to 102:
[0050] Step 101: Acquire a two-dimensional image.
[0051] In step 101, the two-dimensional image includes at least one human head, which may include a face, hair, and facial features, so that the head posture recognition model can identify the head's features. The two-dimensional image can be a color image or a black-and-white image. The two-dimensional image can be captured by a camera module of a monitoring device, or the monitoring device can receive images captured by a connected camera device.
[0052] Step 102: Input the two-dimensional image into a head posture recognition model, identify the head features in the two-dimensional image, and output a rotation matrix of the head features, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using at least a two-dimensional image of a head as a sample and the rotation matrix of the sample as a label.
[0053] In step 101, the acquired two-dimensional image is input into a head posture recognition model. This model is capable of identifying head features in the two-dimensional image and directly obtaining the rotation matrix of the head features in the two-dimensional image. The rotation matrix is used to represent the rotation angle of the head features. In other words, the input of the head posture recognition model is a two-dimensional image, and the output is the rotation matrix of the head features. The two-dimensional image and the rotation matrix have a direct mapping relationship, eliminating the need for complex transformations to obtain the rotation matrix of the head features. The head posture recognition model is trained and generated using at least a two-dimensional head image as a sample and the rotation matrix corresponding to the two-dimensional image as a label. In other words, the neural network model is trained using supervised learning, thereby building a more robust and semantically sound model.
[0054] The rotation matrix is a way to represent the rotation angle of the human head. It is a 3×3 matrix. The rotation matrix includes three vectors. The three vectors are mutually orthogonal space vectors. They describe the space vectors of the three main axes (x-axis, y-axis, and z-axis), representing the directions of the three main axes respectively. The properties of the rotation matrix are shown in the following formula 1:
[0055]
[0056] Among them, R is the rotation matrix, and the three vectors of the rotation matrix are: r1=[r 11 ,r 21 ,r 31 ],r2=[r 12 ,r 22 ,r 32 ] and r3=[r 13 ,r 23 ,r 33 ]. From the above formula 1, we can see that the three vectors of the rotation matrix are perpendicular to each other and are all unit vectors.
[0057] The above-mentioned head posture recognition method of this embodiment inputs a two-dimensional image including a head into a trained head posture recognition model. After recognition by the head posture recognition model, the rotation matrix corresponding to the head features in the two-dimensional image can be directly obtained, and the rotation matrix is used to represent the rotation angle of the head features. Compared with the related art that uses Euler angles to represent the rotation angle of the head features, this technical solution can avoid the problem of large differences between the Euler angles regressed from head features of similar angles due to discontinuity caused by universal joint deadlock, can effectively improve the recognition accuracy, and have more semantic features; and the head posture recognition model in this technical solution is constructed by training using multiple two-dimensional images including heads as samples and the rotation matrices of the samples as labels, which can also further improve the recognition accuracy; using the head posture recognition model to construct a direct mapping relationship between the two-dimensional image and the rotation matrix, the rotation angle of the head features can be quickly obtained, thereby improving the recognition efficiency.
[0058] Because this head posture recognition method is suitable for applications such as traffic monitoring, a monitoring device installed in a vehicle can obtain a two-dimensional image of the head posture of the driver driving the vehicle and execute this head posture recognition method to determine the driver's head posture while driving the vehicle and whether the driver is focused on driving. In an exemplary embodiment of the present application, after the head posture recognition model outputs the rotation matrix of the head features, the method further includes: outputting a warning signal if the rotation matrix exceeds a set rotation matrix range.
[0059] Among them, the set rotation matrix range can be the range of head rotation angles that characterizes the driver's concentration on driving. That is to say, if the output rotation matrix is within the set rotation range, the driver's head posture is relatively facing the front of the driving field of view, which can indicate that the driver is concentrating on driving; if the output rotation matrix exceeds the set rotation range, the driver's head posture is not facing the front of the driving field of view, for example, lowering the head, facing away from the field of view, or looking to the side, etc., which can indicate that the driver is not concentrating on driving. At this time, a reminder signal is output to remind the driver to concentrate on driving and avoid traffic accidents.
[0060] In an exemplary embodiment of the present application, the reminder signal may be a voice signal. For example, a continuous warning sound effect may be output. Another example is a warning sound effect that outputs a human voice.
[0061] In an exemplary embodiment of the present application, the two-dimensional image may be acquired in real time, that is, to monitor in real time whether the driver in the car is focused on driving. In another exemplary embodiment of the present application, the two-dimensional image may be acquired periodically.
[0062] In another exemplary embodiment of the present application, after the monitoring device is started, when the working time reaches a set time length, the above-mentioned head posture recognition method is executed. Generally, the monitoring device is started when the vehicle is started. The startup of the monitoring device indicates that the current vehicle is in use. The working time of the monitoring device can also indicate the driving time of the vehicle, which can indicate the driving time of the driver. When the working time reaches the set time length, for example, 1 hour, 2 hours, etc., it can indicate that the driver's driving time is 1 hour or 2 hours. The driver is prone to fatigue driving and may be in a drowsy state. At this time, the monitoring device can obtain a two-dimensional image including the driver's head and input it into the head posture recognition model to obtain the rotation matrix of the driver's head, that is, the rotation angle, to determine whether the driver is dozing off with his head down. If so, a reminder signal is output to remind the driver to stay awake and avoid causing a traffic accident.
[0063] The following describes in detail how the head posture recognition model identifies and obtains the rotation matrix of the head features in the two-dimensional image after the two-dimensional image is input into the head posture recognition model. Figure 2 This is a flow chart of a head posture recognition model recognition process shown in an exemplary embodiment of the present application. The head posture recognition model shown in this embodiment includes a ResNet (Residual Network) residual network layer, a deconvolution layer and an integration layer, such as Figure 2 As shown, the head posture recognition model recognition process specifically includes the following steps 201 to 203:
[0064] Step 201: Use the residual network layer to extract head features from the two-dimensional image to obtain a network layer with a first size and a first number of channels.
[0065] In step 201, the residual network layer can extract the head features of the two-dimensional image, extract the core features that can characterize the head, and obtain a network layer with a first size and a first number of channels. The first size and the first number of channels can be pre-set.
[0066] In an exemplary embodiment of the present application, the residual network is ResNet-34. Using a 34-layer residual network can ensure the real-time prediction of the head posture recognition model.
[0067] Step 202: Deconvolve the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels.
[0068] In step 201, the deconvolution layer can perform multiple deconvolutions on the network layer obtained by feature extraction from the residual network layer to obtain a feature layer of a second size and a second number of channels. The multiple deconvolution operations can sample the obtained feature layer to a scale that can achieve a clear mapping with the input two-dimensional image.
[0069] In an exemplary embodiment of the present application, the first size is smaller than the second size, and the number of the first channels is greater than the number of the second channels.
[0070] In an exemplary embodiment of the present application, when the deconvolution layer performs a deconvolution operation, the head posture recognition model further includes a BatchNorm (Batch Normalization) operation and a ReLU (Rectified Linear Unit) operation.
[0071] Step 203 , simulating the feature layer into a three-dimensional cube, integrating the three axes of the three-dimensional cube using the integration layer to obtain three vectors of the rotation matrix of the head feature in the two-dimensional image, and outputting the recognition result.
[0072] In step 203, the multi-channel feature layer can be simulated as a three-dimensional cube. The multi-channel feature layer obtained after deconvolution is simulated as a three-dimensional cube. The three axes (x-axis, y-axis, and z-axis) of the three-dimensional cube are integrated to obtain a vector, which includes three axial sub-vectors. The three sub-vectors x, y, and z of a vector are: x is obtained by the yz surface integral of the cube, y is obtained by the xz surface integral of the cube, and z is obtained by the xy surface integral of the cube, thereby obtaining a vector.
[0073] Since the rotation matrix includes three vectors, in an exemplary embodiment of the present application, the number of the three-dimensional cubes simulated by the feature layer is three. Then, the three axes of the three three-dimensional cubes can be integrated respectively to obtain the first vector, the second vector and the third vector of the rotation matrix, so that the human head posture recognition model outputs the rotation matrix corresponding to the two-dimensional image.
[0074] The rotation matrix has the property shown in Formula 1, that is, the three vectors of the rotation matrix are mutually perpendicular and are all unit vectors. Therefore, as long as two of the vectors of the rotation matrix are known, the third vector can be obtained. In another exemplary embodiment of the present application, the feature layer is simulated as two three-dimensional cubes. Then, the three axes of the two three-dimensional cubes are integrated to obtain the first and second vectors of the rotation matrix. The third vector of the rotation matrix is calculated based on the first and second vectors. The head posture recognition model thus outputs the rotation matrix corresponding to the two-dimensional image. This can further improve recognition efficiency.
[0075] In an exemplary embodiment of the present application, the first vector of the rotation matrix is defined as the straight line vector from the center of the human head to the tip of the nose, and the second vector is defined as the straight line vector from the center of the human head to the top of the head. In an exemplary embodiment of the present application, the center of the three-dimensional cube is the center of the human head. This facilitates integration to obtain the rotation matrix.
[0076] The above embodiment is described below by taking an example. Figure 3 This is a flow chart of the process of inputting a two-dimensional image into a head posture recognition model for recognition as shown in an exemplary embodiment of this application. Figure 3 As shown, the two-dimensional image 30 is a color image with a size of 256*256 including head features. The two-dimensional image 30 is input into the head posture recognition model 300, wherein the head posture recognition model 300 includes: a ResNet-34 residual network layer 310, a deconvolution layer 320 and an integration layer (not shown). After the two-dimensional image 30 passes through the ResNet-34 residual network layer 310, a network layer 31 with a size of 8*8 and 512 channels is obtained. After deconvolution by the deconvolution layer 320, a feature layer 32 with a size of 16*16 and 512 channels is obtained. Further deconvolution is performed to obtain a feature layer 33 with a size of 32*32 and 256 channels. Further deconvolution is performed to obtain a feature layer 34 with a size of 64*64 and 128 channels. In this case, the number of channels 128 can be considered as 64*2, that is, the scale of the feature layer 34 is 64*64*64*2. In this way, the feature layer 34 can be simulated as two 64*64*64 three-dimensional cubes. The integration layer integrates the three axes of the two three-dimensional cubes, obtaining three values respectively. For the two cubes, six values are obtained, thus obtaining two vectors. According to the properties of the rotation matrix, only two of the vectors need to be known to obtain the third vector, thus obtaining the rotation matrix of the two-dimensional image 30.
[0077] This application also provides a method for constructing a head posture recognition model. Figure 4 FIG. 1 is a flow chart of a method for constructing a head posture recognition model according to an exemplary embodiment of the present application. Figure 4 As shown, the method for constructing a head posture recognition model includes the following steps 401 to 402:
[0078] Step 401: Acquire a plurality of two-dimensional images at least including a human head and a rotation matrix corresponding to each of the two-dimensional images.
[0079] Step 402 : Using the two-dimensional image as a sample and the rotation matrix of the sample as a label, the set neural network model is trained to construct the head posture recognition model.
[0080] In this embodiment, multiple two-dimensional images including a human head and the rotation matrices corresponding to the two-dimensional images are obtained as training data, wherein the rotation matrices are used as labels, and the sample-label pair is used to perform supervised learning on the set neural network model to construct a neural network model that can accurately recognize the posture of the human head.
[0081] The training process of the set neural network model is described in detail below. Figure 5 This is a flow chart of a method for constructing a head posture recognition model as shown in an exemplary embodiment of the present application. The set neural network model includes a ResNet residual network layer, a deconvolution layer, and an integration layer. Figure 5 As shown, the method for constructing a head posture recognition model includes the following steps 501 to 504:
[0082] Step 501: Use the residual network layer to extract head features from the sample to obtain a network layer with a first size and a first number of channels.
[0083] Step 502: Deconvolve the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels.
[0084] Step 503 : Simulate the feature layer into a three-dimensional cube, and use the integration layer to integrate the three axes of the three-dimensional cube to obtain a prediction result of the rotation matrix of the head feature in the two-dimensional image sample.
[0085] Steps 501 to 503 are substantially the same as the related techniques in steps 201 to 203 above, and will not be described in detail here. The difference is that the obtained rotation matrix is a prediction result.
[0086] Step 504: Use a loss function to obtain a loss value between the prediction result and the corresponding label, and train the set neural network model based on the loss value.
[0087] In step 504, according to Formula 1 of the rotation matrix, any vector in the rotation matrix is a unit vector, i.e., the vector's modulus is 1. Therefore, in a three-dimensional cube, the vector's position should be on a sphere centered at the center of the cube. Therefore, the prediction result of the rotation matrix vector obtained by three-dimensional cube integration, i.e., the prediction of the three subvectors x, y, and z of the vector, should be a prediction of the position on the sphere. However, the network output data is noisy, and the maximum value at a certain point cannot be used as the final prediction standard for x, y, and z. Therefore, x, y, and z are obtained by integrating the cube, i.e., x is obtained by the yz surface integral of the cube, y is obtained by the xz surface integral of the cube, and z is obtained by the xy surface integral of the cube. The rotation matrix vector obtained by integration is the prediction result, and a loss function is then used to obtain the loss value between the prediction result and the true value of the corresponding label. The set neural network model is trained based on the loss value, i.e., the neural network model is trained through backpropagation, thereby constructing a head posture recognition model.
[0088] In an exemplary embodiment of the present application, the loss function is L1loss. L1loss has a globally better gradient conduction property and can train a better head posture recognition model.
[0089] The following is a detailed description of how to obtain the rotation matrix corresponding to the two-dimensional image. Figure 6A This is a flow chart of a process for obtaining a rotation matrix corresponding to a two-dimensional image, as shown in an exemplary embodiment of the present application. Figure 6B This is a schematic diagram of an exemplary embodiment of the present application showing a method of constructing a three-dimensional model using a two-dimensional image. Figure 6A As shown, in an exemplary embodiment of the present application, the step of obtaining the rotation matrix corresponding to the two-dimensional image specifically includes steps 601 to 602:
[0090] Step 601: construct a corresponding three-dimensional model using the two-dimensional image.
[0091] Step 602: Obtain a rotation matrix corresponding to the two-dimensional image based on the three-dimensional model.
[0092] In this embodiment, Figure 6B As shown, the 2D image includes a human head. Key points of the head can be obtained from the 2D image, and the head features and key points can be used to construct a 3D model corresponding to the head in the 2D image. Key points can be characteristic points of facial features or facial contours. After constructing the 3D model, the rotation matrix of the head can be calculated based on the acquired 3D facial data.
[0093] Since the three-dimensional face data involved in the rotation matrix is difficult to collect and annotate, and to obtain the corresponding rotation matrix of the two-dimensional images of different heads, it is necessary to construct a three-dimensional model corresponding to the head, which easily leads to a small number of training samples. In order to obtain a large number of samples, in an exemplary embodiment of the present application, the method further includes:
[0094] Acquiring one of the two-dimensional images;
[0095] Data enhancement preprocessing is performed on the two-dimensional image to obtain a plurality of the two-dimensional images as samples.
[0096] In this embodiment, a single two-dimensional image can be used to perform data enhancement preprocessing on the image to obtain multiple training samples, thereby increasing the amount of training data and being able to quickly train a better head posture recognition model.
[0097] In an exemplary embodiment of the present application, the data augmentation preprocessing includes at least one of the following: image cropping, color conversion, resizing, and image rotation. In other words, a single preprocessing method can be used to increase the number of training samples, or a combination of preprocessing methods can be used to increase the number of training samples. Each resulting training sample is different, thereby increasing the amount of training data.
[0098] The following is a detailed description of the above-mentioned data enhancement preprocessing methods:
[0099] Image cropping is to crop a two-dimensional image into images of different sizes. Due to the change in image size, the position of the head in the two-dimensional image can be changed, thereby forming different training samples. Although the image cropping process does not change the rotation matrix of the head in the two-dimensional image, it enhances the data input to the network.
[0100] Color transformation is to change the overall color of the two-dimensional image, for example, reducing or increasing the brightness, reducing or increasing the contrast, transforming the color image into a black and white image, a grayscale image, etc., thereby forming different training samples. The color transformation process is similar to image cropping. It does not change the rotation matrix of the human head in the two-dimensional image, but enhances the data input to the network.
[0101] Scaling is to enlarge or reduce the size of a two-dimensional image. It is different from image cropping. Scaling can enlarge or reduce the head in the two-dimensional image, or in other words, change the resolution of the head, thereby forming different training samples. The scaling process is similar to image cropping and color transformation. It does not change the rotation matrix of the head in the two-dimensional image, but enhances the network input data.
[0102] Image rotation involves rotating a 2D image, for example, counterclockwise, clockwise, or even mirroring. This can change the rotation matrix of the head in the 2D image, essentially enhancing the labels in the training samples and creating different training samples. Furthermore, there's no need to construct multiple 3D models to obtain the rotation matrix. Instead, a 3D model can be constructed using the unprocessed 2D image, and the rotation matrix of the 2D image can be obtained. This matrix can then be used to determine the rotation matrix of the rotated 2D image.
[0103] For example, the coordinates of a point in the original two-dimensional image are (x, y), and the coordinates of a point mapped to the two-dimensional image after rotation by angle θ are (x′, y′). The two coordinates satisfy the following formula 2:
[0104]
[0105] Expressed as a rotation matrix, the following formula 3 is satisfied:
[0106]
[0107] in, is the rotation matrix of the original two-dimensional image, is the rotation matrix of the two-dimensional image after rotation by angle θ.
[0108] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of the device and the electronic device to which it is applied.
[0109] This application also provides a head posture recognition device, Figure 7 This is a schematic diagram of a head posture recognition device according to an exemplary embodiment of the present application. Figure 7 As shown, the head posture recognition device 70 includes:
[0110] An acquisition module 710 is configured to acquire a two-dimensional image;
[0111] The output module 720 is configured to input the two-dimensional image into a head posture recognition model, identify the head features in the two-dimensional image, and output the rotation matrix of the head features, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using at least a two-dimensional image of a human head as a sample and the rotation matrix of the sample as a label.
[0112] In an exemplary embodiment of the present application, the head posture recognition model includes a ResNet residual network layer, a deconvolution layer, and an integration layer;
[0113] The output module includes:
[0114] a network layer acquisition submodule, configured to extract head features from the two-dimensional image using the residual network layer to obtain a network layer of a first size and a first number of channels;
[0115] a feature layer acquisition submodule, configured to perform deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels;
[0116] The rotation matrix acquisition submodule is configured to simulate the feature layer into a three-dimensional cube, and use the integration layer to integrate the three axes of the three-dimensional cube to obtain three vectors of the rotation matrix of the human head feature in the two-dimensional image.
[0117] In an exemplary embodiment of the present application, the number of the three-dimensional cubes simulated as the feature layer is two, and the rotation matrix acquisition submodule includes:
[0118] a vector acquisition unit configured to integrate the three axial directions of the two three-dimensional cubes to obtain a first vector and a second vector of the rotation matrix;
[0119] A vector calculation unit is configured to calculate a third vector of the rotation matrix according to the first vector and the second vector.
[0120] In an exemplary embodiment of the present application, the number of the three-dimensional cubes simulated as the feature layer is three, and the rotation matrix acquisition submodule includes:
[0121] The vector acquisition unit is configured to integrate the three axial directions of the three three-dimensional cubes to obtain the first vector, the second vector and the third vector of the rotation matrix.
[0122] In an exemplary embodiment of the present application, the first vector is a straight line vector from the center of the human head to the tip of the nose, and the second vector is a straight line vector from the center of the human head to the top of the head.
[0123] In an exemplary embodiment of the present application, the center of the three-dimensional cube is the center of the human head.
[0124] In an exemplary embodiment of the present application, the residual network layer is ResNet-34.
[0125] In an exemplary embodiment of the present application, the apparatus further includes:
[0126] The reminder signal output module is configured to output a reminder signal after the head posture recognition model outputs the rotation matrix of the head feature if the rotation matrix exceeds a set rotation matrix range.
[0127] This application also provides a device for constructing a head posture recognition model. Figure 8 This is a schematic diagram of a device for constructing a head posture recognition model according to an exemplary embodiment of the present application. Figure 8 As shown, the head posture recognition model building device 80 includes:
[0128] A first acquisition module 810 is configured to acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0129] The construction module 820 is configured to train the set neural network model using the two-dimensional image as a sample and the rotation matrix of the sample as a label to construct the head posture recognition model.
[0130] In an exemplary embodiment of the present application, the set neural network model includes a ResNet residual network layer, a deconvolution layer and an integration layer;
[0131] The building blocks include:
[0132] a network layer acquisition submodule, configured to extract head features from the sample using the residual network layer to obtain a network layer of a first size and a first number of channels;
[0133] a feature layer acquisition submodule, configured to perform deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels;
[0134] a prediction result acquisition submodule configured to simulate the feature layer into a three-dimensional cube, and integrate the three axes of the three-dimensional cube using the integration layer to obtain a prediction result of the rotation matrix of the head feature in the two-dimensional image sample;
[0135] The training submodule is configured to use a loss function to obtain a loss value between the prediction result and the corresponding label, and train the set neural network model based on the loss value.
[0136] In an exemplary embodiment of the present application, the loss function is L1loss.
[0137] In an exemplary embodiment of the present application, the apparatus further includes:
[0138] A second acquisition module is configured to acquire one of the two-dimensional images;
[0139] The enhancement preprocessing module is configured to perform data enhancement preprocessing on the two-dimensional image to obtain a plurality of the two-dimensional images as samples.
[0140] In an exemplary embodiment of the present application, the data enhancement preprocessing includes at least one of the following: image cropping, color transformation, size scaling, and image rotation.
[0141] In an exemplary embodiment of the present application, the acquisition module further includes:
[0142] A first acquisition submodule is configured to acquire a two-dimensional image including at least a human head;
[0143] a construction submodule, configured to construct a corresponding three-dimensional model using the two-dimensional image;
[0144] The second acquisition submodule is configured to acquire a rotation matrix corresponding to the two-dimensional image based on the three-dimensional model.
[0145] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0146] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying any creative work.
[0147] The present application also provides a computer device, Figure 9 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present application. Figure 9 As shown, the computer device 90 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented:
[0148] Acquire a two-dimensional image;
[0149] The two-dimensional image is input into a head posture recognition model, the head features in the two-dimensional image are identified, and a rotation matrix of the head features is output, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image including at least a head as a sample and the rotation matrix of the sample as a label.
[0150] This application also provides another computer device, Figure 10 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present application. Figure 10 As shown, the computer device 100 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented:
[0151] Acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0152] The two-dimensional image is used as a sample and the rotation matrix of the sample is used as a label to train the set neural network model to construct the head posture recognition model.
[0153] It should be understood that the computer device that executes the head posture recognition method of the above embodiment and the computer device that executes the head posture recognition model construction method of the above embodiment can be the same computer device or different computer devices, and this application does not make specific limitations.
[0154] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:
[0155] Acquire a two-dimensional image;
[0156] The two-dimensional image is input into a head posture recognition model, the head features in the two-dimensional image are identified, and a rotation matrix of the head features is output, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image including at least a head as a sample and the rotation matrix of the sample as a label.
[0157] The present application also provides another computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:
[0158] Acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images;
[0159] The two-dimensional image is used as a sample and the rotation matrix of the sample is used as a label to train the set neural network model to construct the head posture recognition model.
[0160] It should be understood that the computer-readable storage medium for executing the head posture recognition method of the above embodiment and the computer-readable storage medium for executing the head posture recognition model construction method of the above embodiment can be the same computer-readable storage medium or different computer-readable storage media, and this application does not make specific limitations.
[0161] The embodiments of the present application may take the form of a computer program product implemented on one or more readable media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0162] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0163] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0164] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
[0165] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.
[0166] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A method for recognizing head posture, characterized in that: The method comprises: Acquire a two-dimensional image; Input the two-dimensional image into a head posture recognition model, identify the head features in the two-dimensional image, and output a rotation matrix of the head features, where the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using at least a two-dimensional image including a head as a sample and the rotation matrix of the sample as a label; the rotation matrix includes three mutually perpendicular unit vectors, the first vector of the rotation matrix is a straight line vector from the center of the head to the tip of the nose, and the second vector of the rotation matrix is a straight line vector from the center of the head to the top of the head; The head posture recognition model includes a ResNet residual network layer, a deconvolution layer and an integration layer; The step of obtaining a rotation matrix of head features in the two-dimensional image after the two-dimensional image is input into a head posture recognition model comprises: Extracting head features from the two-dimensional image using the residual network layer to obtain a network layer with a first size and a first number of channels; Performing deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels; The feature layer is simulated into a three-dimensional cube, and the three axes of the three-dimensional cube are integrated using the integration layer to obtain three vectors of the rotation matrix of the human head feature in the two-dimensional image; the center of the three-dimensional cube is the center of the human head.
2. The head posture recognition method according to claim 1, characterized in that: The number of the three-dimensional cubes simulating the feature layer is two, and the step of integrating the three axial directions of the three-dimensional cubes includes: Integrating the three axes of the two three-dimensional cubes to obtain a first vector and a second vector of the rotation matrix; A third vector of the rotation matrix is calculated based on the first vector and the second vector.
3. The head posture recognition method according to claim 1, characterized in that: The number of the three-dimensional cubes simulating the feature layer is three, and the step of integrating the three axial directions of the three-dimensional cubes includes: The three axes of the three three-dimensional cubes are integrated to obtain the first vector, the second vector and the third vector of the rotation matrix.
4. The head posture recognition method according to claim 1, characterized in that: The residual network layer is ResNet-34.
5. The head posture recognition method according to claim 1, characterized in that: After the head posture recognition model outputs the rotation matrix of the head feature, the method further includes: If the rotation matrix exceeds the set rotation matrix range, a warning signal is output.
6. A method for constructing a head posture recognition model, characterized in that: The method comprises: Acquire multiple two-dimensional images of at least a human head and a rotation matrix corresponding to each of the two-dimensional images; the rotation matrix includes three mutually perpendicular unit vectors, the first vector of the rotation matrix is a straight line vector from the center of the human head to the tip of the nose, and the second vector of the rotation matrix is a straight line vector from the center of the human head to the top of the head; Using the two-dimensional image as a sample and the rotation matrix of the sample as a label, the set neural network model is trained to construct the head posture recognition model; The set neural network model includes a ResNet residual network layer, a deconvolution layer and an integration layer; The steps of training the set neural network model include: Extracting head features from the sample using the residual network layer to obtain a network layer with a first size and a first number of channels; Performing deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels; The feature layer is simulated into a three-dimensional cube, and the three axes of the three-dimensional cube are integrated using the integration layer to obtain a prediction result of the rotation matrix of the human head feature in the two-dimensional image sample; the center of the three-dimensional cube is the center of the human head; A loss function is used to obtain a loss value between the prediction result and the corresponding label, and the set neural network model is trained based on the loss value.
7. The method for constructing a head posture recognition model according to claim 6, wherein: The loss function is L1loss.
8. The method for constructing a head posture recognition model according to claim 6, wherein: The method further comprises: Acquiring one of the two-dimensional images; Data enhancement preprocessing is performed on the two-dimensional image to obtain a plurality of the two-dimensional images as samples.
9. The method for constructing a head posture recognition model according to claim 8, wherein: The data enhancement preprocessing includes at least one of the following: Image cropping, color conversion, resizing, and image rotation.
10. The method for constructing a head posture recognition model according to claim 6, wherein: The step of obtaining the rotation matrix corresponding to the two-dimensional image includes: constructing a corresponding three-dimensional model using the two-dimensional image; A rotation matrix corresponding to the two-dimensional image is obtained based on the three-dimensional model.
11. A head posture recognition device, characterized in that: The device comprises: an acquisition module configured to acquire a two-dimensional image; An output module is configured to input the two-dimensional image into a head posture recognition model, identify head features in the two-dimensional image, and output a rotation matrix of the head features, wherein the rotation matrix is used to represent the rotation angle of the head features; wherein the head posture recognition model is a model generated by training using a two-dimensional image of at least a head as a sample and the rotation matrix of the sample as a label; the rotation matrix includes three mutually perpendicular unit vectors, the first vector of the rotation matrix is a straight line vector from the center of the head to the tip of the nose, and the second vector of the rotation matrix is a straight line vector from the center of the head to the top of the head; The head posture recognition model includes a ResNet residual network layer, a deconvolution layer and an integration layer; The step of obtaining a rotation matrix of head features in the two-dimensional image after the two-dimensional image is input into a head posture recognition model comprises: Extracting head features from the two-dimensional image using the residual network layer to obtain a network layer with a first size and a first number of channels; Performing deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels; The feature layer is simulated into a three-dimensional cube, and the three axes of the three-dimensional cube are integrated using the integration layer to obtain three vectors of the rotation matrix of the human head feature in the two-dimensional image; the center of the three-dimensional cube is the center of the human head.
12. A device for constructing a head posture recognition model, characterized in that: The device comprises: an acquisition module configured to acquire a plurality of two-dimensional images including at least a human head and a rotation matrix corresponding to each of the two-dimensional images; the rotation matrix comprising three mutually perpendicular unit vectors, the first vector of the rotation matrix being a straight line vector from the center of the human head to the tip of the human nose, and the second vector of the rotation matrix being a straight line vector from the center of the human head to the top of the human head; A construction module is configured to train a set neural network model using the two-dimensional image as a sample and the rotation matrix of the sample as a label to construct the head posture recognition model; The set neural network model includes a ResNet residual network layer, a deconvolution layer and an integration layer; The steps of training the set neural network model include: Extracting head features from the sample using the residual network layer to obtain a network layer with a first size and a first number of channels; Performing deconvolution on the network layer several times using the deconvolution layer to obtain a feature layer of a second size and a second number of channels; The feature layer is simulated into a three-dimensional cube, and the three axes of the three-dimensional cube are integrated using the integration layer to obtain a prediction result of the rotation matrix of the human head feature in the two-dimensional image sample; the center of the three-dimensional cube is the center of the human head; A loss function is used to obtain a loss value between the prediction result and the corresponding label, and the set neural network model is trained based on the loss value.
13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the head posture recognition method described in any one of claims 1 to 5 are implemented.
14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for constructing a head posture recognition model described in any one of claims 6 to 10 are implemented.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the head posture recognition method described in any one of claims 1 to 5 are implemented.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for constructing a head posture recognition model described in any one of claims 6 to 10 are implemented.