A head pose estimation method and device based on structured light images

Abstract

The application provides a head pose estimation method and device based on a structured light image, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a to-be-tested structured light image containing a target face; performing face positioning on the to-be-tested structured light image to obtain positioning information; cutting the to-be-tested structured light image according to the positioning information to obtain a to-be-tested structured light face image containing only a face region; and inputting the to-be-tested structured light face image into a trained head pose estimation model to obtain a head pose estimation result of the target face. Through implementation of the application, the structured light image has the comprehensive characteristics of near-infrared images and depth images, can provide more abundant image information for head pose judgment, and effectively improves the accuracy of head pose estimation.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a head pose estimation method and apparatus based on structured light images. Background Technology

[0002] Head pose estimation technology refers to the technique of inferring the three rotation angles (yaw, pitch, and roll) of the head in three-dimensional space from a face image. Current mainstream head pose estimation techniques typically use RGB or near-infrared images for facial landmark detection. A set of three-dimensional coordinates for these landmarks is defined, and then the pose of the camera coordinate system relative to the world coordinate system is calculated. This yields an approximate transformation matrix from the projection of the three-dimensional landmarks to the detected landmarks. Finally, the rotation matrix is ​​extracted to obtain the head pose data. Because this technique relies on the landmark features of two-dimensional images, errors in landmark feature recognition can lead to limited accuracy in the estimated head pose data. Summary of the Invention

[0003] This application provides a head pose estimation method and apparatus based on structured light images, which can at least solve the problem of low accuracy of head pose estimation based on two-dimensional images in related technologies.

[0004] The first aspect of this application provides a head pose estimation method based on structured light images, comprising: acquiring a structured light image to be tested containing a target face; performing face localization on the structured light image to be tested to obtain localization information; cropping the structured light image to be tested according to the localization information to obtain a structured light face image to be tested containing only the face region; and inputting the structured light face image to be tested into a trained head pose estimation model to obtain a head pose estimation result of the target face.

[0005] A second aspect of this application provides a head pose estimation device based on structured light images, comprising: an acquisition module for acquiring a structured light image to be tested containing a target face; a localization module for performing face localization on the structured light image to be tested to obtain localization information; a cropping module for cropping the structured light image to be tested according to the localization information to obtain a structured light face image to be tested containing only the face region; and an estimation module for inputting the structured light face image to be tested into a trained head pose estimation model to obtain a head pose estimation result of the target face.

[0006] A third aspect of this application provides a depth camera, including: a transmitter, a receiver, a memory, and a processor; the transmitter is used to emit a structured light beam toward the region where the target face is located; the receiver is used to acquire the structured light beam reflected by the target face to generate a structured light image to be tested, and to transmit the structured light image to be tested to the processor; the processor is used to execute a computer program stored in the memory; when the processor executes the computer program, it implements the steps of the head pose estimation method based on structured light images provided in the first aspect of this application.

[0007] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the head pose estimation method based on structured light images provided in the first aspect of this application.

[0008] The fifth aspect of this application provides a computer program product that, when run on a computer, implements the steps of the head pose estimation method based on structured light images provided in the first aspect of this application.

[0009] As can be seen from the above, this application utilizes structured light images to train a corresponding head pose estimation model, obtaining a test structured light image containing the target face; performs face localization on the test structured light image to obtain localization information; crops the test structured light image based on the localization information to obtain a test structured light face image containing only the face region; and inputs the test structured light face image into the trained head pose estimation model to obtain the head pose estimation result of the target face. Through the implementation of this application's scheme, structured light images possess the combined characteristics of near-infrared images and depth images, providing richer image information for head pose determination, effectively improving the accuracy of head pose estimation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the structure of a depth camera provided in one embodiment of this application;

[0011] Figure 2 A basic flowchart of a head pose estimation method provided in an embodiment of this application is shown below;

[0012] Figure 3 This is a schematic diagram of a speckle structured light image and a speckle structured light face image provided in an embodiment of this application;

[0013] Figure 4 This is a schematic diagram of head pose information provided in one embodiment of this application;

[0014] Figure 5 A schematic diagram of the structure of a head pose estimation model provided in an embodiment of this application;

[0015] Figure 6 A schematic diagram of another head pose estimation model provided in an embodiment of this application;

[0016] Figure 7 A schematic diagram of the program modules of a head pose estimation device provided in an embodiment of this application. Detailed Implementation

[0017] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The following will describe in detail, with reference to the accompanying drawings, a head pose estimation method and apparatus based on structured light images according to embodiments of this application.

[0019] Existing head pose estimation methods primarily estimate the head pose of a face in RGB or IR images. Traditional head pose calculation methods solve the 2D-to-3D correspondence problem by estimating some facial landmarks and using an averaged head model. Some newer research utilizes deep learning-based methods, treating the angle information of facial orientation as a multi-task regression or classification model. This model takes an RGB or IR image containing the face as input and outputs three Euler angles representing the head orientation. Compared to traditional head pose estimation algorithms, this method does not rely on facial landmarks and achieves higher accuracy and better performance.

[0020] However, in some embodiments, if a depth camera is to have head pose estimation capabilities, it needs to be able to acquire RGB images or send the acquired IR images under floodlights to the depth camera's processor. However, in scenarios where the depth camera only acquires structured light images, adding a separate thread for RGB or IR images to obtain head pose information for a face image would incur significant unnecessary costs. Furthermore, in some special applications, the depth camera may be required to send only the acquired structured light images to the processor, which then performs head pose estimation and other image processing functions. In such scenarios, conventional head pose estimation methods cannot be used to output the estimation results.

[0021] This application provides a head pose estimation method based on structured light images, applied to a depth camera, such as... Figure 1The diagram shows a schematic of a depth camera according to an embodiment of this application. The depth camera 1 mainly includes: a transmitter 11, a receiver 12, a processor 13, and a memory 14. The transmitter 11 is used to emit a structured light beam towards the area where the target face is located. The receiver 12 is used to acquire the structured light beam reflected by the target face to generate a structured light image to be tested, and to transmit the structured light image to be tested to the processor 13. The processor 13 is connected to the transmitter 11 and the receiver 12 respectively, and is used to execute a computer program stored in the memory 14. When the processor 13 executes the computer program, it implements the following flow of a head pose estimation method based on structured light images: First, acquire a structured light image to be tested containing the target face; then, perform face localization on the structured light image to be tested to obtain localization information; crop the structured light image to be tested according to the localization information to obtain a structured light face image to be tested containing only the face region; finally, input the structured light face image into the trained head pose estimation model to obtain the head pose estimation result of the target face.

[0022] In some embodiments, the transmitter 11 includes a light source and a patterned optical element. The light source emits an infrared beam of a specific wavelength (e.g., 850nm, 940nm), and the patterned optical element modulates the infrared beam (e.g., replicates, splits, diffracts, etc.) to emit a structured light beam towards the area where the target face is located. In practical applications, depending on the encoding method, the patterned optical element can emit a speckle structured light beam or a stripe structured light beam. This embodiment preferably uses a speckle structured light beam, resulting in a denser speckle distribution in the acquired speckle structured light image. This denser distribution can more accurately reflect depth information. Therefore, the depth camera using a patterned optical element that emits a speckle structured light beam in the transmitter 11 achieves higher accuracy in head pose estimation. It should be noted that in this embodiment, the light source can include a single VCSEL or a VCSEL array, and the patterned optical element can include any one of a diffractive optical element (DOE), a microlens array (MLA), or a mask; no single limitation is made here.

[0023] In some embodiments, the receiver 12 includes at least one image sensor for receiving the light beam reflected back from the face and generating a structured light image. Preferably, the light beam emitted by the light source of the transmitter 11 is infrared light, then the image sensor configured in the receiver 12 is an infrared image sensor. It should be noted that the receiver 12 also includes a lens and a filter, wherein the lens is used to receive the light beam reflected back from the user and focus it onto the corresponding pixel of the image sensor, and the filter is used to filter stray light so that only the light beam emitted by the transmitter 11 enters the receiver 12, thereby obtaining an accurate structured light image.

[0024] In some embodiments, processor 13 is used to acquire the structured light image generated by receiver 12. Processor 13 can also use a depth calculation algorithm to calculate the distortion of the structured light image to obtain the position and depth information of the object. Processor 13 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), neural network chips, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0025] In some embodiments, the memory 14 can be an internal storage unit, such as a hard disk or RAM; the memory can also be an external storage device, such as an external hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory can include both internal storage units and external storage devices, and can also be used to temporarily store data that has been output or will be output. It should be noted that when the processor is a neural network chip, the depth camera 1 in this embodiment may not include a memory. In practical applications, whether the depth camera 1 needs to use memory to store the corresponding computer program depends on the type of processor.

[0026] This application embodiment also provides a computer-readable storage medium, which may be disposed in the aforementioned depth camera. The computer-readable storage medium may be the aforementioned... Figure 1 The memory in the illustrated embodiment.

[0027] In some embodiments, the computer-readable storage medium stores a computer program that, when executed by a processor, implements the aforementioned head pose estimation method based on structured light images. Furthermore, the computer-readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0028] An embodiment of this application also provides a computer program product containing instructions, which, when run on a computer or processor, can implement the aforementioned head pose estimation method based on structured light images.

[0029] like Figure 2 The diagram shown is a basic flowchart of a head pose estimation method based on structured light images provided in an embodiment of this application. This head pose estimation method can be derived from... Figure 1 The depth camera operation includes the following steps:

[0030] Step 201: Obtain the structured light image of the target face.

[0031] Specifically, in a real head pose estimation scenario, the transmitter emits a structured light beam toward the actual face in the scene, and the receiver generates a structured light image to be tested based on the structured light beam reflected from the actual face. The structured light image to be tested can preferably be a speckle structured light image.

[0032] Step 202: Perform face localization on the structured light image to be tested to obtain localization information.

[0033] Specifically, in practical applications, the image generated by the receiving end is not limited to the face region. The image regions outside the face region are usually redundant image information. In this embodiment, to reduce the computational load and improve accuracy of subsequent head pose estimation, face localization is performed on the structured light image after acquiring the structured light image containing the face. This localization locates the face region in the image, resulting in a structured light face image. Face localization refers to checking whether an image contains a face and further determining the position and size of the face. Typically, a rectangular bounding box is used to mark the face region. In some applications, the rectangular bounding box itself serves as the localization information.

[0034] In one embodiment, face localization of the acquired structured light image includes: comparing the structured light image with a reference structured light image to identify discrepancies in the structured light image that are significantly different from those in the reference structured light image in terms of speckle or stripe patterns; further extracting features from the discrepancies in the structured light image; and marking the face position based on the symmetry and three-dimensional features of the face (the height of the nose is greater than that of the cheek).

[0035] In one embodiment, face localization of the acquired structured light image includes: calculating the depth information of the structured light image using a depth calculation algorithm to obtain a depth image; inputting the depth image into a face localization model to output a depth image with face location information; and marking a rectangle or marking the pixels of the face location in the structured light image based on the face location information in the depth image.

[0036] Step 203: Crop the structured light image to be tested based on the positioning information to obtain a structured light face image to be tested that only contains the face region.

[0037] Specifically, when a rectangular bounding box is used to represent the face region for face localization, the structured light image can be obtained by cropping the structured light image based on this bounding box identifier. Taking a speckle structured light image as an example, such as... Figure 3 The image shown is a schematic diagram of a speckle structured light image and a speckle structured light face image provided in this embodiment. Figure 3 In this image, 'a' represents a structured light image containing the target face, and 'b' represents a structured light face image cropped from the speckle structured light image in 'a' based on the positioning information.

[0038] In some embodiments, face localization is performed in an image by marking pixels belonging to the face instead of using bounding boxes to identify the face. The marked pixels are the localization information. The structured light image is cropped based on the marked pixels, which can also be understood as extracting the marked pixels from the structured light image to obtain a structured light face image containing only the face region.

[0039] Step 204: Input the structured light face image to be tested into the trained head pose estimation model to obtain the head pose estimation result of the target face.

[0040] Specifically, this embodiment incorporates a head pose estimation model tailored to the characteristics of structured light images. The head pose estimation results in this embodiment include three types of head pose information: yaw, roll, and pitch. Figure 4 The diagram shown is a schematic diagram of head posture information provided in this embodiment.

[0041] like Figure 5 The diagram shown is a structural schematic of a head pose estimation model provided in an embodiment of this application. This head pose estimation model includes N sequentially connected convolutional processing modules (such as...). Figure 5 The model consists of convolutional processing modules 1, 2, ..., N, and a fully connected layer, where N is an integer greater than 1. Each convolutional processing module includes a convolutional layer and an activation layer. Correspondingly, the steps described above for inputting the structured light face image to be tested into the trained head pose estimation model to obtain the head pose estimation result of the target face include: inputting the structured light face image to be tested into the trained head pose estimation model; sequentially using the convolutional layers of each convolutional processing module for feature extraction and dimensionality reduction; mapping the output features of the convolutional layers to a high-dimensional nonlinear region using the activation layers of the convolutional processing modules; and performing global convolution on the output features of the last convolutional processing module using a fully connected layer to obtain the head pose estimation result of the target face.

[0042] Specifically, in this embodiment, for multiple sequentially connected convolutional processing modules, the output of the previous convolutional processing module serves as the input of the next convolutional processing module. For each convolutional processing module, the output of its convolutional layer serves as the input of the activation layer. Furthermore, the output of the activation layer of the last convolutional processing module serves as the input of the fully connected layer. It should be noted that the convolutional layer extracts features and reduces dimensionality of the image through convolutional operations. While ensuring rich feature extraction, it filters redundant information and reduces the amount of image processing. The activation layer adds non-linearity to the model, mapping features to a high-dimensional non-linear interval for interpretation, thus solving problems that linear models cannot address. In addition, the main function of the fully connected layer is to fuse all features together to obtain global features, facilitating result regression.

[0043] This embodiment is in Figure 5 Based on the provided head pose estimation model, it also provides, for example, Figure 6 The diagram shows another head pose estimation model. In this embodiment, the number of channels in multiple convolutional processing modules increases sequentially according to their connection order. The head pose estimation model also includes a branch convolutional processing module with the same structure as the aforementioned convolutional processing module. The input of the branch convolutional processing module corresponds to the output of the first convolutional processing module, and the number of channels in the branch convolutional processing module is equal to the number of channels in the first convolutional processing module. Please refer again. Figure 6 In a preferred embodiment, the head pose estimation model includes three sequentially connected convolutional processing modules with 12, 16, and 64 channels respectively. The branch convolutional processing module has 12 channels. The output of the first convolutional processing module is used as the input of the branch convolutional processing module. In this embodiment, the Concat module can be used to fuse the outputs of the branch convolutional processing module and the last convolutional processing module among the multiple sequentially connected convolutional processing modules. The fused features are used as the input of the fully connected layer.

[0044] It should also be noted that, in the preferred embodiment, the size of the last convolutional processing module in the multiple sequentially connected convolutional processing modules of the above embodiment is 3×1, and the size of the other convolutional processing modules and the branch convolutional processing modules is 4×4.

[0045] Accordingly, the steps described above, which utilize a fully connected layer to perform global convolution on the output features of the last convolutional processing module to obtain the head pose estimation result of the target face, include: performing a fusion operation on the output features of the last convolutional processing module and the output features of the branch convolutional processing modules to obtain fused features; and using a fully connected layer to perform global convolution on the fused features to obtain the head pose estimation result of the target face.

[0046] The head pose estimation model provided in this embodiment has the characteristics of having a small number of parameters and relatively low computational load in the model application stage. Therefore, it has low resource requirements for the hardware on which the model is deployed, that is, it has the characteristics of low deployment cost and is highly feasible to deploy on low power consumption and low computing power devices.

[0047] In an optional implementation of this embodiment, the training process of the head pose estimation model includes: inputting training sample data from the training sample set into the head pose estimation model to be trained to obtain prediction data; calculating the training loss value based on the label data of the prediction data and the training sample data; if the training loss value does not meet the preset model convergence condition, adjusting the configuration parameters of the head pose estimation model to be trained based on the training loss value to obtain an intermediate model; using the intermediate model as the head pose estimation model to be trained again, and continuing to execute the steps of inputting training sample data from the training sample set into the head pose estimation model to be trained and obtaining prediction data, until the training loss value meets the model convergence condition, thus obtaining the trained head pose estimation model.

[0048] Specifically, the head pose estimation model in this embodiment is pre-trained based on a specific training sample set, which includes multiple training sample data. During model training, the model output data corresponding to the training sample data, i.e., the prediction data, is obtained. Then, the loss value is calculated for the prediction data and the pre-labeled data of the training sample data. In this embodiment, the loss value is used to measure the difference between the network's training output and the real label. Next, it is determined whether the loss value meets the model convergence condition. The model convergence condition can be that the loss value is less than or equal to a preset loss value threshold, i.e., the loss value threshold is used as the critical value for model convergence. If the loss value during the training process does not meet the model convergence condition, the backpropagation algorithm can be used to calculate the gradient of each parameter of the network and update the model's configuration parameters. Iterative training continues until the loss value meets the model convergence condition, and the model converges, finally obtaining the trained head pose estimation model.

[0049] Further, in an optional embodiment of this example, the process of acquiring the training sample set includes: sending a beam emission command to the transmitting end; wherein, the beam emission command is used to control the transmitting end to emit a pre-trained structured light beam towards the area where the face is located, and the pre-trained structured light beam is the same as the structured light beam corresponding to the structured light image to be tested; acquiring a structured light image generated by the receiving end based on the pre-trained structured light beam reflected by the face; performing face localization on the structured light image to obtain localization information; cropping the structured light image according to the localization information to obtain a structured light face image containing only the face area; labeling the structured light face image with tag data to obtain training sample data; and generating a training sample set based on multiple training sample data. The face localization method is the same as the method for performing face localization on the acquired structured light image in the above embodiment, and will not be described again here.

[0050] Specifically, in this embodiment, the transmitting end projects a pre-trained structured light beam that is identical to the structured light beam corresponding to the structured light image to be tested onto the target area. Then, the receiving end collects the beam reflected back from the target area to generate a structured light image containing the target face. Next, the head pose of the collected structured light image is labeled (Yaw, Roll, and Pitch) to obtain training sample data. The above process is repeated multiple times to obtain a large amount of training sample data as a training sample set.

[0051] Furthermore, in an optional implementation of this embodiment, the step of generating a training sample set based on multiple training sample data includes: performing data augmentation processing on the multiple training sample data to obtain processed sample data; inputting the multiple training sample data and all processed sample data into a preset sample data simulation model to generate simulated training sample data; and combining all training sample data and simulated training sample data to generate a training sample set.

[0052] Specifically, in practical applications, the accuracy of a model is closely related to the training process, and the training of a high-precision model cannot be separated from training with a large number of training samples. This embodiment considers the large workload of manually labeling training samples and therefore uses sample simulation to expand the training sample data. In this embodiment, data augmentation processing, such as horizontal flipping, rotation, and cutout, can be performed on the labeled original training sample data. Next, based on a preset data simulation model, sample data simulation is performed on the original training sample data and the processed sample data to generate multiple simulated training sample data sets. Finally, the training sample data and the simulated training sample data are combined to obtain a training sample set with a significantly increased sample size.

[0053] Furthermore, in another optional embodiment of this example, the loss function of the head pose estimation model to be trained can be expressed as:

[0054]

[0055] Where x = |yaw_pre - yaw_gt| + |pitch_pre - pitch_gt| + |roll_pre - roll_gt|, yaw_pre represents the predicted yaw angle data, yaw_gt represents the labeled yaw angle data, pitch_pre represents the predicted pitch angle data, pitch_gt represents the labeled pitch angle data, roll_pre represents the predicted roll angle data, and roll_gt represents the labeled roll angle data.

[0056] Specifically, the loss function in this embodiment can be implemented using Smooth L1 Loss. Smooth L1 Loss is a piecewise function that combines the advantages of both L1 Loss and L2 Loss. It uses a smooth L2 Loss when |x| < 1 and a stable L1 Loss when |x| ≥ 1. L1 Loss has the following advantages: stable gradient values, resulting in smooth training; less susceptible to outliers (dirty data), treating all data equally. L2 Loss has the following advantages: smooth and differentiable; large gradients when x is large, leading to fast convergence; small gradients when x is small, making it easier to converge to extreme points.

[0057] Figure 7 A head pose estimation device based on structured light images is provided as an embodiment of this application. This head pose estimation device can be used to implement the head pose estimation method based on structured light images in the aforementioned embodiments. The head pose estimation device mainly includes:

[0058] The acquisition module 701 is used to acquire a structured light image of the target human face.

[0059] The positioning module 702 is used to perform face localization on the structured light image under test and obtain positioning information;

[0060] The cropping module 703 is used to crop the structured light image to be tested according to the positioning information to obtain a structured light face image to be tested that only contains the face region.

[0061] The estimation module 704 is used to input the structured light face image into the trained head pose estimation model to obtain the head pose estimation result of the target face; wherein, the head pose estimation result includes yaw angle, roll angle and pitch angle.

[0062] In some embodiments of this example, the head pose estimation device further includes a model training module, configured to: input training sample data from the training sample set into the head pose estimation model to be trained to obtain prediction data; calculate a training loss value based on the label data of the prediction data and the training sample data; if the training loss value does not meet the preset model convergence condition, adjust the configuration parameters of the head pose estimation model to be trained based on the training loss value to obtain an intermediate model; use the intermediate model as the head pose estimation model to be trained again, and continue to execute the function of inputting training sample data from the training sample set into the head pose estimation model to be trained to obtain prediction data, until the training loss value meets the model convergence condition, and obtain the trained head pose estimation model.

[0063] In some embodiments of this example, the head pose estimation device further includes: a sample generation module, configured to: send a beam emission command to the transmitting end; wherein the beam emission command is used to control the transmitting end to emit a pre-trained structured light beam toward the area where the face is located, the pre-trained structured light beam being the same as the structured light beam corresponding to the structured light image to be tested; acquire a structured light image generated by the receiving end based on the pre-trained structured light beam reflected from the face; label the structured light image with tag data to obtain training sample data; and generate a training sample set based on multiple training sample data.

[0064] In some embodiments of this example, when the sample generation module performs the function of generating a training sample set based on multiple training sample data, it is specifically used to perform data augmentation processing on the multiple training sample data to obtain processed sample data; input the multiple training sample data and all processed sample data into a preset sample data simulation model to generate simulated training sample data; and combine all training sample data and simulated training sample data to generate a training sample set.

[0065] In some embodiments of this example, the head pose estimation model includes N sequentially connected convolutional processing modules and a fully connected layer. Each convolutional processing module includes a convolutional layer and an activation layer. Accordingly, the estimation module is specifically used to: input a structured light face image into the trained head pose estimation model; sequentially perform feature extraction and dimensionality reduction using the convolutional layers of each convolutional processing module; map the output features of the convolutional layers to a high-dimensional nonlinear region using the activation layers of the convolutional processing modules; and perform global convolution on the output features of the last convolutional processing module using the fully connected layer to obtain the head pose estimation result of the target face.

[0066] In some implementations of this embodiment, the number of channels in the N convolutional processing modules increases sequentially according to the connection order. The head pose estimation model also includes a branch convolutional processing module, where the input of the branch convolutional processing module corresponds to the output of the first convolutional processing module, and the number of channels in the branch convolutional processing module is equal to the number of channels in the first convolutional processing module. Accordingly, when the estimation module performs the function of globally convolving the output features of the last convolutional processing module using a fully connected layer to obtain the head pose estimation result of the target face, it specifically performs the following: fusion operation on the output features of the last convolutional processing module and the output features of the branch convolutional processing module to obtain fused features; and global convolution of the fused features using a fully connected layer to obtain the head pose estimation result of the target face.

[0067] It should be noted that the head pose estimation methods in the foregoing embodiments can all be implemented based on the head pose estimation device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the head pose estimation device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] Based on the technical solution of the above embodiments of this application, a structured light image containing the target face is acquired; face localization is performed on the structured light image to obtain localization information; the structured light image to be tested is cropped according to the localization information to obtain a structured light face image containing only the face region; the structured light face image is input into a trained head pose estimation model to obtain the head pose estimation result of the target face. Through the implementation of this solution, the structured light image possesses the combined characteristics of near-infrared and depth images, providing richer image information for head pose determination, effectively improving the accuracy of head pose estimation, and exhibiting superior performance in close-range and large-angle scenes for head pose estimation.

[0069] It should be noted that the apparatuses and methods disclosed in the several embodiments provided in this application can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0070] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0071] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0072] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0073] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0074] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0075] The above is a description of the head pose estimation method and apparatus based on structured light images provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A head pose estimation method based on structured light images, characterized in that, include: Acquire a structured light image containing the target face, wherein the structured light image is a speckle structured light image; Face localization is performed on the structured light image to be tested to obtain localization information; The structured light image to be tested is cropped according to the positioning information to obtain a structured light face image to be tested that only contains the face region. The structured light face image to be tested is input into the trained head pose estimation model to obtain the head pose estimation result of the target face; wherein, the head pose estimation model includes N sequentially connected convolutional processing modules, a branch convolutional processing module connected to the first convolutional processing module, a fully connected layer, and a Concat module. Each convolutional processing module includes a convolutional layer and an activation layer. The number of channels of the N convolutional processing modules increases sequentially according to the connection order. The number of channels of the branch convolutional processing module is equal to the number of channels of the first convolutional processing module. The Concat module is used to receive the output features of the branch convolution processing module and the output features of the last convolution processing module among the N sequentially connected convolution processing modules, and perform a fusion operation to obtain fused features; the fully connected layer is used to perform global convolution on the fused features to obtain the head pose estimation result of the target face.

2. The head pose estimation method according to claim 1, characterized in that, The training process of the head pose estimation model includes: The training sample data is input into the head pose estimation model to be trained to obtain prediction data. Calculate the training loss value based on the predicted data and the label data of the training sample data; If the training loss value does not meet the preset model convergence condition, the configuration parameters of the head pose estimation model to be trained are adjusted according to the training loss value to obtain an intermediate model. The intermediate model is then used again as the head pose estimation model to be trained. The steps of inputting training sample data from the training sample set into the head pose estimation model to be trained and obtaining prediction data are continued until the training loss value meets the model convergence condition, thus obtaining the trained head pose estimation model.

3. The head pose estimation method according to claim 2, characterized in that, The loss function of the head pose estimation model to be trained is expressed as: ； in, x =|yaw_pre-yaw_gt|+|pitch_pre-pitch_gt|+|roll_pre-roll_gt|, where yaw_pre represents the predicted yaw angle, yaw_gt represents the labeled yaw angle, pitch_pre represents the predicted pitch angle, pitch_gt represents the labeled pitch angle, roll_pre represents the predicted roll angle, and roll_gt represents the labeled roll angle.

4. The head pose estimation method according to claim 2, characterized in that, The process of obtaining the training sample set includes: Send a beam emission command to the transmitter; wherein, the beam emission command is used to control the transmitter to emit a pre-trained structured light beam toward the area where the face is located, and the pre-trained structured light beam is the same as the structured light beam corresponding to the structured light image to be tested; Acquire a structured light image generated by the receiving end based on the pre-trained structured light beam reflected from the face; Face localization is performed on the structured light image to obtain localization information; The structured light image is cropped based on the positioning information to obtain a structured light face image containing only the face region; The structured light face image is labeled with tags to obtain training sample data; The training sample set is generated based on multiple training sample data.

5. The head pose estimation method according to claim 4, characterized in that, The process of generating the training sample set based on multiple training sample data includes: Data augmentation processing is performed on multiple training sample data to obtain processed sample data; Multiple training sample data and all processed sample data are input into a preset sample data simulation model to generate simulated training sample data. The training sample set is generated by combining all the training sample data and the simulated training sample data.

6. A head pose estimation device based on structured light images, characterized in that, include: The acquisition module is used to acquire a structured light image to be tested containing the target face, wherein the structured light image to be tested is a speckle structured light image; The positioning module is used to perform face localization on the structured light image under test to obtain positioning information; The cropping module is used to crop the structured light image to be tested according to the positioning information to obtain a structured light face image to be tested that only contains the face region. An estimation module is used to input the structured light face image into a trained head pose estimation model to obtain the head pose estimation result of the target face. The head pose estimation model includes N sequentially connected convolutional processing modules, a branch convolutional processing module connected to the first convolutional processing module, a fully connected layer, and a Concat module. Each convolutional processing module includes a convolutional layer and an activation layer. The number of channels of the N convolutional processing modules increases sequentially according to the connection order. The number of channels of the branch convolutional processing module is equal to the number of channels of the first convolutional processing module. The Concat module is used to fuse the output features of the branch convolutional processing module and the output features of the last convolutional processing module among the N sequentially connected convolutional processing modules to obtain fused features; the fully connected layer is used to perform global convolution on the fused features to obtain the head pose estimation result of the target face.

7. A depth camera, characterized in that, Includes the transmitter, receiver, memory, and processor; The transmitting end is used to emit a structured light beam toward the area where the target face is located; The receiving end is used to acquire the structured light beam reflected by the target face to generate a structured light image to be tested, and to transmit the structured light image to be tested to the processor; The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the head pose estimation method based on structured light images according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the head pose estimation method based on structured light images as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, When the computer program product is run on a computer, it implements the steps of the head pose estimation method based on structured light images as described in any one of claims 1 to 5.

Images