Face model reconstruction method, device, terminal device and storage medium

Abstract

The present application relates to a facial model reconstruction method, apparatus, terminal device, and storage medium. The method comprises: acquiring facial data from multiple frames of captured images in a video sequence; and aligning the facial data in each frame with a standard pose model. Compared to iterative alignment using captured images between different frames, this method reduces the amount of computational data required for the iterative calculation and eliminates the need for successful alignment each time, thereby improving the robustness of the model reconstruction. After aligning the facial data with the standard pose model, the facial pose corresponding to the facial data is obtained; keyframe images are determined from the multiple frames of captured images based on the facial pose; and a facial model is reconstructed based on the facial data in the keyframe images.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, terminal device, and storage medium for reconstructing a face model. Background Art

[0002] As animation and game development continue to gain widespread application in fields such as animation and game development, the need for facial model reconstruction is increasing. However, existing 3D facial reconstruction based on RGDB cameras requires multiple acquisition devices to capture facial data from different angles and poses, requiring extensive hardware.

[0003] Although there is also an implementation method for collecting facial data of different postures through a single acquisition device, the alignment method for facial pose estimation in this implementation method is to iteratively align and fuse the depth point cloud of the current frame with the depth point cloud of the previous frame. This alignment method has a large amount of computation and needs to ensure that each alignment is successful, which has poor robustness for facial model reconstruction. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides a face model reconstruction method, apparatus, terminal device and storage medium.

[0005] In a first aspect, the present application provides a method for reconstructing a face model, comprising:

[0006] Acquire face data from multiple frames of captured images in a video sequence;

[0007] Registering the facial data with a standard posture model to estimate the facial posture corresponding to the facial data;

[0008] Determining key frame images from the multiple frames of collected images based on the facial posture, wherein the facial postures corresponding to the key frame images are all different;

[0009] A face model is generated based on the face data in the key frame image.

[0010] In a second aspect, the present application provides a face model reconstruction device, comprising:

[0011] An acquisition module is used to acquire face data from multiple frames of captured images in a video sequence;

[0012] a registration module, configured to register the facial data with a standard posture model and estimate the facial posture corresponding to the facial data, wherein the facial postures corresponding to the key frame images are all different;

[0013] A determination module, configured to determine a key frame image from the plurality of acquired frames based on the facial posture;

[0014] A generation module is used to generate a face model based on the face data in the key frame image.

[0015] In a third aspect, the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0016] Acquire face data from multiple frames of captured images in a video sequence;

[0017] Registering the facial data with a standard posture model to estimate the facial posture corresponding to the facial data;

[0018] Determining a key frame image from the multiple frames of collected images based on the facial posture;

[0019] A face model is generated based on the face data in the key frame image.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] Acquire face data from multiple frames of captured images in a video sequence;

[0022] Registering the facial data with a standard posture model to estimate the facial posture corresponding to the facial data;

[0023] Determining a key frame image from the multiple frames of collected images based on the facial posture;

[0024] A face model is generated based on the face data in the key frame image.

[0025] Based on the above-mentioned facial model reconstruction method, facial data is obtained from multiple frames of acquired images in a video sequence. The facial data in each acquired frame is then registered with a standard pose model. Compared to iterative registration using acquired images between different frames, this method reduces the amount of computational data required for the iterative calculation and eliminates the need for successful registration every time, thereby improving the robustness of the model reconstruction. After registering the facial data with the standard pose model, the facial pose corresponding to the facial data is obtained. Based on the facial pose, key frames are identified from the multiple acquired frames. A facial model is then reconstructed based on the facial data in the key frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 A diagram illustrating an application environment of a face model reconstruction method according to an embodiment;

[0029] Figure 2 1 is a flow chart of a face model reconstruction method according to an embodiment;

[0030] Figure 3 A schematic diagram of a model coordinate system in one embodiment;

[0031] Figure 4 A schematic diagram of program modules of a face model reconstruction device according to one embodiment;

[0032] Figure 5 2 is a structural diagram of a terminal device in one embodiment. DETAILED DESCRIPTION

[0033] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] Figure 1 FIG. 1 is an application environment diagram of a face model reconstruction method in an embodiment. Figure 1 , the face model reconstruction method is applied to a face model reconstruction system. The face model reconstruction system includes an acquisition device 110 and a processor 120. The acquisition device 110 and the processor 120 can be integrated into the same device, the acquisition device 110 and the processor 120 can be independently set respectively, or the processor 120 is integrated into the acquisition device 110, and the acquisition device 110 and the processor 120 can be connected via a wired or wireless manner. The acquisition device 110 can specifically be any device that can acquire depth maps and color maps, such as a 3D camera or a 3D sensor, and can specifically be an RGBD device. The processor 120 can be implemented as an independent processor 120 or a processor 120 cluster consisting of multiple processors 120.

[0035] In one embodiment, Figure 2This is a flow chart of a face model reconstruction method in one embodiment, referring to Figure 2 , provides a face model reconstruction method. This embodiment mainly applies this method to the above Figure 1 Taking the acquisition device 110 in FIG. 1 as an example, the face model reconstruction method specifically includes the following steps:

[0036] Step S210: acquiring face data from multiple frames of captured images in a video sequence.

[0037] In this embodiment, a video sequence indicates scan data acquired by a single acquisition device 110 through continuous scanning. The video sequence comprises multiple frames of continuous captured images. Feature extraction is performed on the captured images, and the extracted facial features are matched against preset facial features. When the matching degree reaches a feature threshold, a face is determined to be present in the captured image, and the data associated with the facial features is used as facial data. This eliminates the need to utilize multiple acquisition devices 110 to capture facial data in different poses, reduces the hardware required, and thus lowers the cost of reconstructing the facial model.

[0038] Step S220 , registering the facial data with a standard posture model to estimate the facial posture corresponding to the facial data.

[0039] In this embodiment, the standard posture model is used to indicate a model obtained after deep training based on multiple facial images with the faces facing the acquisition device 110. The standard posture model is used as a reference, and the facial posture of the facial data relative to the standard posture model is determined based on the alignment result between the facial data and the standard posture model. The facial posture is used to indicate the deflection angle of the facial features relative to the acquisition device 110 and the state of the facial features of the face. The deflection angle is used to indicate whether the face is in a head deflection state such as side face, head tilted up or head down. The state of the facial features is used to indicate whether the eyes are open or closed, the mouth is open or closed, etc.

[0040] Facial data is obtained from multiple frames of captured images in a video sequence, and the facial data in each frame of the captured image is aligned with the standard posture model. Compared with iterative alignment using captured images between different frames, this reduces the amount of computational data in the iterative calculation part, and does not require that each alignment be successful, thereby improving the robustness of the model reconstruction.

[0041] Step S230 : determining key frame images in the multiple frames of collected images based on the facial posture, wherein the facial postures corresponding to the key frame images are all different.

[0042] In this embodiment, since there may be multiple frames of captured images in the video sequence with repeated facial postures, that is, there may be multiple frames of captured images with the same facial posture, in order to avoid a large amount of repeated facial data from participating in the model reconstruction calculation, screening is performed based on the facial posture, and frame images with repeated facial postures are filtered out, and frame images with different facial postures are retained as key frame images.

[0043] Step S240: Generate a face model based on the face data in the key frame image.

[0044] In this embodiment, when the processor 120 is integrated into the acquisition device 110, the processor 120 in the acquisition device 110 uses data related to facial features in key frame images indicating different facial postures to perform model reconstruction, thereby realizing facial model reconstruction.

[0045] In one embodiment, facial data is obtained from multiple frames of captured images in a video sequence, including: obtaining a pixel-aligned color image and a depth image based on the video sequence; when a face is detected in the color image, determining position information of the face in the color image; determining key point coordinates related to facial features in the color image; and using the key point coordinates and a depth point cloud in the depth image as facial data in the captured image.

[0046] Specifically, the acquired image includes a color image and a depth image. Each pixel value in the color image is divided into three primary color categories of R, G, and B. Each primary color category directly determines the intensity of its primary color, that is, each pixel value in the color image is used to indicate the color value of the pixel. Each pixel value in the depth image is the actual distance between the acquisition device 110 and the acquisition object. Pixel alignment means that there is a one-to-one correspondence between the pixel points of the color image and the pixel points of the depth map. It should be noted that the acquisition device 110 can be a depth camera and a color camera, which are used to respectively acquire depth images and color images. The depth camera and the color camera can be set independently of each other or integrated. The depth camera can be any one of a structured light camera, a TOF camera, a binocular camera, etc., and is not limited here.

[0047] Based on the implementation process of feature extraction and comparison for face detection in the above embodiment, when a face is detected in a color image, the position information of the face in the color image is determined. For example, the position information indicates that the face is located in the upper left corner of the color image. This is equivalent to first circling the area containing the face in the color image, and then using points related to facial features in the area as key points, such as a point indicating the chin, a point indicating the tip of the nose, a point indicating the left corner of the eye, a point indicating the right corner of the eye, a point indicating the left corner of the mouth, and a point indicating the right corner of the mouth, etc., and using the coordinates of the key points in the color image as key point coordinates, which are two-dimensional coordinates. The depth point cloud in the depth image refers to the point cloud related to facial features in the depth image, and the key point coordinates and the depth point cloud used to indicate facial features are used as face data.

[0048] In one embodiment, after determining the coordinates of the key points related to facial features in the color image, the method further includes: determining the distance between the face and the acquisition device 110 based on the position information and the depth information of the depth map; when the distance value is within a preset distance range, executing the step of using the key point coordinates and the depth point cloud in the depth map as facial data in a single-frame acquired image.

[0049] Specifically, the depth information is used to indicate the distance between the depth point cloud in the depth map and the acquisition device 110. By combining the position information of the face in the color map and the depth information of the depth map, the distance between the face and the acquisition device 110 can be accurately determined. The preset distance range is used to indicate the optimal acquisition distance of the acquisition device 110. That is, when the distance between the user and the acquisition device 110 falls within the preset distance range, the display effect of the acquired image acquired by the acquisition device 110 is optimal, avoiding the user being too close or too far from the acquisition device 110, thereby ensuring the reliability of the acquired image. Therefore, only when it is determined through the position information and depth information that the user's face is at an appropriate distance from the acquisition device 110, the key point coordinates and the depth point cloud in the depth map will be used as facial data for the subsequent model reconstruction process, and an accurate facial model can be reconstructed based on reliable facial data.

[0050] In one embodiment, facial data is aligned with a standard posture model to estimate the facial posture corresponding to the facial data, including: determining the rotation matrix of the depth point cloud relative to the standard posture model based on key point coordinates; converting the rotation matrix into a rotation angle in a model coordinate system; wherein the rotation angle is used to indicate the facial posture.

[0051] Specifically, all facial features in the depth map are aligned and registered with the corresponding facial features in the standard posture model based on the key point coordinates. For example, the point cloud representing the tip of the nose in the depth point cloud is aligned and registered with the data related to the tip of the nose in the standard posture model, so as to obtain the transformation relationship of the point cloud representing the tip of the nose relative to the standard posture model. Specifically, the rotation matrix can be determined by the solvePnP function or the solvePnPRansac function, so that the transformation relationship of the depth point cloud relative to the standard posture model can be obtained according to the alignment results of each facial feature with the standard posture model. The transformation relationship is represented by a rotation matrix, which contains the transformation information of rotation and translation. In general, only the transformation information of rotation is used. Further, the rotation matrix is converted into Euler angles, that is, the coordinate system of the depth point cloud is converted from the camera coordinate system to the model coordinate system, and the rotation angle in the model coordinate system is obtained. The camera coordinate system refers to a coordinate system whose origin is established on the acquisition device 110, with the camera of the acquisition device 110 as the geometric center. The model coordinate system refers to a coordinate system whose origin is established at the geometric center of the model, such as Figure 3 As shown, the positive direction of the Z axis is the direction pointed outward from the tip of the nose, the Y axis is vertically upward, the X axis is determined by the right hand system, and the rotation angle includes the profile angle, pitch angle and yaw angle. The profile angle is used to indicate the angle of rotation around the Y axis, the pitch angle is used to indicate the angle of rotation around the X axis, and the yaw angle is used to indicate the angle of rotation around the Z axis. The facial posture is represented by the rotation angle, that is, the profile angle, pitch angle and yaw angle.

[0052] In one embodiment, a key frame image is determined in multiple frames of captured images based on facial posture, including: determining the capture state of the captured image based on the side face angle in the rotation angle; when the capture state is not captured, using the captured image as the key frame image.

[0053] Specifically, the side face angle is used to indicate the deflection of the face. The side face angle is queried in the collected data to determine the acquisition status of the collected image corresponding to the side face angle. The acquisition status includes collected and not collected. The collected data includes the collected image and the facial posture of the collected image relative to the standard posture model. The facial posture is represented by the side face angle, pitch angle and roll angle. When the angle value corresponding to the side face angle is queried in the collected data, it means that the collected image corresponding to the side face angle has been collected, and the acquisition status of the collected image is determined to be collected; when the angle value corresponding to the side face angle is not queried in the collected data, it means that the collected image corresponding to the side face angle has not been collected, and the acquisition status of the collected image is determined to be not collected.

[0054] By filtering the acquisition status described above, we can remove some images with the same pose, reducing the amount of computational data required for model reconstruction. Images with the acquisition status "not acquired" are used as keyframe images. That is, multiple keyframe images are acquired images corresponding to different profile angles.

[0055] In one embodiment, before determining the acquisition state of the acquired image based on the side face angle in the rotation angle, the method also includes: determining the frame position of the acquired image in the video sequence; when the frame position of the acquired image is the first frame, determining the orientation state of the facial posture relative to the acquisition device 110; wherein the orientation state includes facing forward and not facing forward; when the orientation state is facing forward, generating indication information.

[0056] Specifically, a video sequence includes multiple frames of continuous captured images, and the frame position is used to indicate the arrangement number of the captured images in the video sequence. The arrangement number can be any character representing the arrangement order, such as a number, an uppercase letter, a lowercase letter, etc. When the arrangement number is represented by a number 1, 2, 3..., N, etc., 1 is used to indicate the captured image at the first frame, and N is used to indicate the captured image at the last frame. When the frame position of the captured image is the first frame, in order to ensure the quality of the captured three-dimensional facial data, it is necessary to ensure that the face in the first frame captured image is facing the capture device 110. Therefore, it is necessary to determine the orientation of the facial posture relative to the capture device 110. The orientation states include facing forward and facing backward. Facing forward means that the face is facing the capture device 110, and facing backward means that there is an angle between the face and the capture device 110, and the face is not facing the capture device 110. When the facial posture in the first frame of the captured image is facing forward relative to the orientation of the capture device 110, the first frame of the captured image and the facial posture of the captured image relative to the standard posture model are saved in the captured data, and indication information is generated. The indication information is used to instruct the capture object corresponding to the facial data to adjust the head posture, that is, to prompt the capture object to turn the head left and right to adjust the side face angle, so as to facilitate the acquisition of captured images with different side face angles.

[0057] In one embodiment, after determining the frame position of the captured image in the video sequence, the method further includes: when the frame position of the captured image is not the first frame, executing the step of determining the capture state of the captured image based on the side face angle in the rotation angle.

[0058] Specifically, when it is determined that the frame position of the captured image is not the first frame, there is no need to execute the above-mentioned step of determining the orientation state of the facial posture relative to the capture device 110, and the step of determining the capture state of the captured image based on the side face angle in the rotation angle is directly executed. That is to say, the captured image of the non-first frame can be judged as a captured image only based on the side face angle.

[0059] In one embodiment, when the acquisition status is not acquired, after the acquired image is used as a key frame image, the method further includes: changing the flag of the face rotation angle range corresponding to the side face angle of the key frame image from the first character to the second character; when the flag of all face rotation angle ranges within the preset range is the second character, executing the step of generating a face model based on the face data in the key frame image.

[0060] Specifically, the face rotation angle range refers to the angle range of the face rotation relative to the acquisition device 110. Each face rotation angle range corresponds to a flag bit, and the flag bit is used to indicate whether a captured image with a side face angle within the face rotation angle range is captured. The flag bit can be represented by any character. The first character is used to indicate that the captured image corresponding to the face rotation angle range is not captured, and the second character is used to indicate that the captured image corresponding to the face rotation angle range is captured. The characters can be numbers, letters, symbols, etc. For example, multiple face rotation angle ranges are [0°, 30°), [30°, 60°), [60°, 90°), and the side face angle is 20°. The flag bit corresponding to the face rotation angle range [0°, 30°) is changed from 0 to 1, and there is no need to repeatedly capture the captured image with a side face angle within [0°, 30°) in the future.

[0061] The preset range refers to the face rotation range required for face model reconstruction, which can be customized according to the face model reconstruction requirements. The preset range includes multiple face rotation angle ranges. For example, if the preset range is [-60°, 60°], the preset range includes [-60°, -30°), [-30°, 0°), [0°, 30°), and [30°, 60°]. A positive angle value indicates that the face is deflected to the right, and a negative angle value indicates that the face is deflected to the left, or vice versa, a positive angle value indicates that the face is deflected to the left, and a negative angle value indicates that the face is deflected to the right. The preset range can also be composed entirely of positive angle values to represent the deflection of the face from right to left.

[0062] When the flag bits corresponding to all face rotation angle ranges within the preset range are all 1, it means that the required images of different side face angles have been collected, and image collection is stopped. The collected images can be used to reconstruct the face model.

[0063] In one embodiment, a face model is generated based on face data in a key frame image, including: determining the pose of a depth point cloud in the key frame image relative to a standard face model; fusing depth point clouds corresponding to multiple key frame images based on the pose to generate a comprehensive point cloud; filtering the comprehensive point cloud to obtain a filtered comprehensive point cloud; and generating a face model represented by grid data based on the filtered comprehensive point cloud.

[0064] Specifically, the pose is used to indicate the transformation relationship between the depth point cloud and the standard face model. Based on the pose, the coordinate system of the depth point cloud can be converted from the camera coordinate system to the model coordinate system. According to the transformation relationship between the key images of each frame and the standard face model, the coordinate system of the depth point cloud of multiple key images is converted to the same coordinate system. In this way, the depth point clouds of multiple key images are fused to obtain a comprehensive point cloud. The comprehensive point cloud is filtered to eliminate noise points to obtain a filtered comprehensive point cloud. The filtered comprehensive point cloud is used to establish a topological structure, and then a mesh model is generated, which is the above-mentioned face model represented by mesh data.

[0065] Figure 2 FIG. 1 is a flow chart of a face model reconstruction method in one embodiment. It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0066] In one embodiment, Figure 4 As shown, a face model reconstruction device is provided, comprising:

[0067] An acquisition module 310 is configured to acquire face data from multiple frames of captured images in a video sequence;

[0068] A registration module 320 is used to register the facial data with a standard posture model and estimate the facial posture corresponding to the facial data;

[0069] A determination module 330 is configured to determine key frame images from the multiple frames of collected images based on the facial posture, wherein the facial postures corresponding to the key frame images are all different;

[0070] The generation module 340 is used to generate a face model based on the face data in the key frame image.

[0071] In one embodiment, the acquisition module 310 is further configured to:

[0072] Acquire a pixel-aligned color image and a depth image based on a video sequence; wherein the acquired image includes a color image and a depth image;

[0073] When a face is detected in the color image, determining the position information of the face in the color image;

[0074] Determining the coordinates of the key points associated with facial features in the color image;

[0075] The key point coordinates and the depth point cloud in the depth map are used as the face data in the captured image.

[0076] In one embodiment, the acquisition module 310 is further configured to:

[0077] Determine the distance between the face and the acquisition device 110 based on the position information and the depth information of the depth map;

[0078] When the distance value is within a preset distance range, the step of using the key point coordinates and the depth point cloud in the depth map as the face data in the single-frame captured image is executed.

[0079] In one embodiment, the registration module 320 is further configured to:

[0080] Determine the rotation matrix of the depth point cloud relative to the standard pose model based on the key point coordinates;

[0081] The rotation matrix is converted into a rotation angle in the model coordinate system; the rotation angle is used to indicate the face pose.

[0082] In one embodiment, the determination module 330 is further configured to:

[0083] Determining a capture state of the captured image based on the profile angle in the rotation angle; wherein the capture state includes captured and not captured;

[0084] When the acquisition status is not acquired, the acquired image is used as the key frame image.

[0085] In one embodiment, the determination module 330 is further configured to:

[0086] Determine the frame position of the captured image in the video sequence;

[0087] When the frame position of the captured image is the first frame, determining the orientation of the face posture relative to the capture device 110; wherein the orientation includes facing forward and facing backward;

[0088] When the orientation state is the forward facing state, instruction information is generated; wherein the instruction information is used to instruct the collection object corresponding to the facial data to adjust the head posture.

[0089] In one embodiment, the determination module 330 is further configured to:

[0090] When the frame position of the acquired image is not the first frame, a step of determining the acquisition state of the acquired image based on the side face angle in the rotation angle is performed.

[0091] In one embodiment, the determination module 330 is further configured to:

[0092] Change the flag of the face rotation angle range of the side face angle corresponding to the key frame image from the first character to the second character;

[0093] When the flag bit of all face rotation angle ranges within the preset range is the second character, the step of generating a face model based on the face data in the key frame image is executed.

[0094] In one embodiment, the generating module 340 is further configured to:

[0095] Determine the pose of the depth point cloud in the key frame image relative to the standard face model;

[0096] Based on the pose fusion of the depth point clouds corresponding to multiple key frame images, a comprehensive point cloud is generated;

[0097] Performing filtering on the integrated point cloud to obtain a filtered integrated point cloud;

[0098] Based on the filtered integrated point cloud, a face model represented by mesh data is generated.

[0099] Figure 5 The internal structure diagram of a terminal device in one embodiment is shown. The terminal device can be Figure 1 The collection device 110 in FIG. Figure 5 As shown, the terminal device includes a processor 120, memory, a network interface, an input device, and a display screen connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium of the terminal device stores an operating system and may also store a computer program. When executed by the processor 120, the computer program enables the processor 120 to implement the facial model reconstruction method. The internal memory may also store a computer program. When executed by the processor 120, the computer program enables the processor 120 to perform the facial model reconstruction method.

[0100] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal device to which the solution of the present application is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0101] In one embodiment, the face model reconstruction device provided by the present application can be implemented in the form of a computer program. Figure 5 The terminal device's memory can store various program modules that constitute the face model reconstruction device, such as: Figure 4 The computer program composed of the illustrated acquisition module 310, registration module 320, determination module 330 and generation module 340 enables the processor 120 to execute the steps of the face model reconstruction method of each embodiment of the present application described in this specification.

[0102] Figure 5 The terminal device shown can be Figure 4 The acquisition module 310 in the illustrated facial model reconstruction apparatus acquires facial data from multiple frames of captured images in a video sequence. The terminal device, through the registration module 320, can register the facial data with a standard pose model to estimate the facial pose corresponding to the facial data. The terminal device, through the determination module 330, can determine keyframe images from the multiple frames of captured images based on the facial pose. The terminal device, through the generation module 340, can generate a facial model based on the facial data in the keyframe images.

[0103] In one embodiment, a terminal device is provided, including a memory, a processor 120, and a computer program stored in the memory and executable on the processor 120. When the processor 120 executes the computer program, the method described in any one of the above embodiments is implemented.

[0104] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor 120, the method described in any of the above embodiments is implemented.

[0105] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware to perform the processes. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0107] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for reconstructing a face model, characterized in that: The method comprises: Obtain pixel-aligned color and depth images based on video sequences; When a face is detected in the color image, determining position information of the face in the color image; wherein the position information is used to indicate an area in the color image where the face is located; Determining the coordinates of key points related to facial features in the position information of the color image; Using the key point coordinates and the depth point cloud in the depth map as face data in the acquired image; wherein the acquired image includes the color map and the depth map; Determining a rotation matrix of the depth point cloud relative to a standard pose model based on the key point coordinates; Converting the rotation matrix into a rotation angle in a model coordinate system; wherein the rotation angle is used to indicate a facial posture; Determining key frame images from the multiple frames of collected images based on the facial posture, wherein the facial postures corresponding to the key frame images are all different; A face model is generated based on the face data in the key frame image.

2. The method according to claim 1, characterized in that After determining the coordinates of key points related to facial features in the position information of the color image, the method further includes: Determining a distance between the face and a capture device based on the position information and the depth information of the depth map; When the value of the distance is within a preset distance range, the step of using the key point coordinates and the depth point cloud in the depth map as the face data in the captured image is performed.

3. The method according to claim 1, characterized in that Determining a key frame image from multiple frames of collected images based on the facial posture includes: Determining a capture state of the captured image based on the side face angle in the rotation angle; wherein the capture state includes captured and not captured; When the acquisition state is not acquired, the acquired image is used as the key frame image.

4. The method according to claim 3, characterized in that Before determining the acquisition state of the acquired image based on the side face angle in the rotation angle, the method further includes: Determining a frame position of the captured image in the video sequence; When the frame position of the captured image is the first frame, determining the orientation state of the facial posture relative to the capture device; wherein the orientation state includes facing forward and facing backward; When the orientation state is the forward facing state, instruction information is generated; wherein the instruction information is used to instruct the collection object corresponding to the facial data to adjust the head posture.

5. The method according to claim 4, characterized in that After determining the frame position of the acquired image in the video sequence, the method further includes: When the frame position of the acquired image is not the first frame, the step of determining the acquisition state of the acquired image based on the side face angle in the rotation angle is performed.

6. The method according to claim 3, characterized in that When the acquisition state is not acquired, after using the acquired image as the key frame image, the method further includes: Changing the flag of the face rotation angle range in which the side face angle corresponding to the key frame image is located from the first character to the second character; When the flag bits of all the face rotation angle ranges within the preset range are the second character, executing the step of generating the face model based on the face data in the key frame image; The first character is used to indicate that the captured image corresponding to the face rotation angle range is not captured, and the second character is used to indicate that the captured image corresponding to the face rotation angle range is captured.

7. The method according to claim 6, characterized in that The step of generating a face model based on the face data in the key frame image includes: Determining the pose of the depth point cloud in the key frame image relative to a standard face model; Fusion of the depth point clouds corresponding to the multiple key frame images based on the posture generates a comprehensive point cloud; Performing filtering on the integrated point cloud to obtain the filtered integrated point cloud; Based on the filtered integrated point cloud, a face model represented by mesh data is generated.

8. A facial model reconstruction device, characterized in that: The device comprises: An acquisition module is configured to acquire a pixel-aligned color image and a depth image based on a video sequence; when a face is detected in the color image, determine position information of the face in the color image; wherein the position information indicates an area of the color image where the face is located; determine coordinates of key points associated with facial features in the position information of the color image; and use the key point coordinates and a depth point cloud in the depth image as facial data in a captured image; wherein the captured image includes the color image and the depth map; A registration module, configured to determine a rotation matrix of the depth point cloud relative to a standard pose model based on the key point coordinates; and convert the rotation matrix into a rotation angle in a model coordinate system; wherein the rotation angle is used to indicate a face pose; A determination module, configured to determine a key frame image in a plurality of acquired frames based on the facial posture, wherein the facial postures corresponding to the key frame images are all different; A generation module is used to generate a face model based on the face data in the key frame image.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. The terminal device according to claim 9, wherein: The terminal device further includes an acquisition device, and the processor uses the acquired image acquired by the acquisition device to execute the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images