Texture mapping method and apparatus based on three-dimensional face reconstruction, device and storage medium

Abstract

The present application relates to the technical field of image processing. Disclosed are a texture mapping method and apparatus based on three-dimensional (3D) face reconstruction, a device and a storage medium. The method comprises: registering a standard 3DMM model with a 3D grid model, and performing region division on the 3D grid model; obtaining a texture image of a target object, and matching 2D face key points of the texture image with 3D face key points of the registered 3DMM model to obtain head posture information of the texture image; and on the basis of whether the head posture information meets preset requirements or not, determining whether to use the texture image for mapping. According to the technical solution, a standard 3DMM model is used to perform region division on a 3D grid model of a target object, and on the basis of head posture information of texture images, a texture image with the head posture meeting preset requirements is selected to perform mapping processing on a corresponding region, thereby making the texture of the mapping more consistent, and constructing a 3D face model having a better visual effect.

Description

Texture mapping method, device, equipment and storage medium based on three-dimensional reconstruction of human face

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 20, 2023, with application number 202310896865X, and invention name “Texture mapping method, device, equipment and storage medium based on three-dimensional reconstruction of human face”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a texture mapping method, apparatus, device and storage medium based on three-dimensional reconstruction of a human face. Background Art

[0003] 3D facial reconstruction uses sensors like cameras to capture image data of real faces and processes it through a processor to create a 3D facial model. However, due to the rich textures of human faces and the variations in expressions and gaze directions across multiple facial images, this can easily lead to texture misalignment, resulting in an unsightly 3D facial model. Technical issues

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present application provide a texture mapping method, apparatus, device and storage medium based on three-dimensional reconstruction of the face.

[0005] To this end, in the first aspect, the present application provides a texture mapping method based on three-dimensional reconstruction of a face, comprising: obtaining a three-dimensional mesh model and a standard 3DMM model of a target object; wherein the standard 3DMM model is preset with a first area and 3D face key points; aligning the standard 3DMM model with the three-dimensional mesh model to obtain a registered 3DMM model; dividing the three-dimensional mesh model into regions according to the positional relationship between the vertices of the registered 3DMM model and the first area to obtain a second area of ​​the three-dimensional mesh model; obtaining a texture image of the target object, and performing face key point detection on the texture image to obtain 2D face key points; matching the 2D face key points with the 3D face key points of the registered 3DMM model to obtain head posture information of the texture image; judging whether the head posture information meets the preset requirements, and if so, mapping the texture image into the second area. Technical Solutions

[0006] The embodiment of the present application divides the area of ​​the three-dimensional mesh model of the target object through the standard 3DMM model, and selects the texture image whose head posture meets the preset requirements according to the head posture information of the texture image to perform mapping processing on the divided area, so as to achieve more consistent texture of the mapping and reconstruct a three-dimensional face model with better visual effect.

[0007] In a possible implementation of the first aspect, 2D facial key points are matched with 3D facial key points of the registered 3DMM model to obtain head posture information of the texture image, including: calculating the projection relationship between the 2D facial key points and the 3D facial key points based on the first coordinate set of the 2D facial key points in the texture camera coordinate system and the second coordinate set of the 3D facial key points in the world coordinate system; and obtaining the head posture information of the texture image based on the projection relationship. Beneficial effects

[0008] The embodiment of the present application converts the coordinates of 2D face key points and 3D face key points in different coordinate systems to obtain head posture information, which can effectively correct the projection pixel error of the texture image.

[0009] In the second aspect, the present application provides a texture mapping device based on three-dimensional reconstruction of a face, including: a model acquisition module, used to obtain a three-dimensional mesh model of a target object and a standard 3DMM model with a preset first area; wherein the standard 3DMM model is preset with a first area and 3D face key points; a model alignment module, used to align the standard 3DMM model with the three-dimensional mesh model to obtain a registered 3DMM model; a region division module, used to divide the three-dimensional mesh model into regions according to the positional relationship between the vertices of the registered 3DMM model and the first area, to obtain a second area of ​​the three-dimensional mesh model; a key point detection module, used to obtain a texture image of the target object, and perform face key point detection on the texture image to obtain 2D face key points; a posture estimation module, used to match the 2D face key points with the 3D face key points of the registered 3DMM model to obtain head posture information of the texture image; a mapping judgment module, used to judge whether the head posture information meets the preset requirements, and if so, map the texture image to the second area.

[0010] In a third aspect, the present application provides a three-dimensional facial reconstruction device, a camera, a processor, and a memory storing a computer program that can be executed by the processor; the camera is used to collect 3D information data and texture images of the target object and send them to the processor; the processor is used to receive the 3D information data and generate a three-dimensional mesh model based on the 3D information data; the processor is also used to implement the steps of the method described in the first aspect when executing the computer program.

[0011] In a fourth aspect, the present application provides an electronic device comprising a processor and a memory storing a computer program; when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0012] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] FIG1 is a schematic block diagram of a structure of a 3D face reconstruction device provided by an embodiment of the present application;

[0016] FIG2 is a flow chart of a texture mapping method based on three-dimensional reconstruction of a face according to an embodiment of the present application;

[0017] FIG3 is a schematic diagram of point cloud data provided by an embodiment of the present application;

[0018] FIG4 is a schematic diagram of a three-dimensional grid model provided in an embodiment of the present application;

[0019] FIG5( a ) is a schematic diagram of a face mesh model including a preset number of vertices provided in an embodiment of the present application;

[0020] FIG5( b ) is a schematic diagram of a face mesh model with vertex numbers marked, provided in an embodiment of the present application;

[0021] FIG5( c ) is a schematic diagram of a face mesh model divided into different regions provided by an embodiment of the present application;

[0022] FIG6 is a schematic diagram of the correspondence between 3D facial key points and 2D facial key points provided by an embodiment of the present application;

[0023] FIG7 is a schematic diagram of the structure of head posture information provided by an embodiment of the present application;

[0024] FIG8 is a schematic block diagram of the structure of a texture map based on three-dimensional reconstruction of a face provided by an embodiment of the present application;

[0025] FIG9 is a schematic block diagram of the structure of an electronic device provided in one embodiment of the present application. Best Mode for Carrying Out the Invention

[0026] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0027] It should be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] References to "some embodiments" and the like in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, the phrase "in some embodiments" appearing in various places in this specification does not necessarily refer to the same embodiment, but rather means "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0029] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined. The term "and / or" as used herein includes any and all combinations of one or more of the relevant listed items.

[0030] 3D facial reconstruction requires mapping a texture image onto a 3D mesh model. This process involves texture mapping the mesh to complete the model. However, existing texture mapping methods fail to consider the semantic continuity of the texture image mapped onto the 3D mesh model, which can easily lead to problems such as texture misalignment and unsightly visual effects in the resulting 3D facial model.

[0031] Based on this, this application uses a three-dimensional deformable face (3DMM) model to divide the area of ​​the three-dimensional mesh model of the target object, and according to the head posture information of the texture image, selects a texture image whose head posture meets the preset requirements to perform mapping processing on the corresponding area, so that the texture of the mapping is more consistent and can optimize the visual effect of the three-dimensional face model.

[0032] Please refer to Figure 1, which is a schematic block diagram of the structure of a three-dimensional facial reconstruction device provided by an embodiment of the present application. The three-dimensional facial reconstruction device includes a camera 12, a processor 13 and a memory 14 connected via a bus 11. The camera 12 is used to collect 3D information data and texture images of the target object, the processor 13 is used to provide computing and control capabilities, and the memory 14 stores a computer program that can be run by the processor 13. The 3D information data can be used to represent the geometric shape of the object, and the 3D information data includes but is not limited to parallax images, depth images and point cloud data. It should be noted that when the processor 13 is a processor in a terminal with computing capabilities such as a host computer, a computer, or a payment device, the camera 12 can also be connected to the processor 13 wirelessly, such as by transmitting the 3D information data to the processor 13 via wifi or Bluetooth.

[0033] Specifically, camera 12 captures 3D information data and a texture image of a target object. Processor 13 obtains the 3D information data and texture image of the target object and invokes a computer program from memory 14 to process the 3D information data to obtain a 3D mesh model. The texture image is then mapped onto the 3D mesh model to construct a 3D model of the target object with texture information. The 3D mesh model acquisition method includes: obtaining 3D information data of the target object, meshing the 3D information data, and obtaining a 3D mesh model.

[0034] In some embodiments, the camera 12 may include a texture camera and a depth camera. The texture camera may be a multispectral camera, an RGB camera, an infrared camera, etc. The depth camera may be a time-of-flight (ToF)-based camera such as an indirect time-of-flight (iToF) or direct time-of-flight (dToF), a binocular vision camera, or a structured light camera, etc.

[0035] In some embodiments, when the depth camera is a structured light camera, the structured light camera includes a projector and a collector, the projector is used to project a structured light beam, and the collector is used to collect the structured light beam reflected by the target object to obtain a structured light image. A matching calculation is performed on the structured light image to obtain a disparity image, and the depth corresponding to each pixel is calculated using the disparity image to obtain a depth image. The depth image is converted to a coordinate system to obtain point cloud data, and the point cloud data is gridded to obtain a three-dimensional mesh model. The matching calculation, disparity image calculation, coordinate system conversion, and gridding processes can be performed by a depth calculation chip within the structured light camera or by a processor outside the structured light camera, and are not limited here.

[0036] In some embodiments, when the depth camera is a ToF camera, the ToF camera includes a light source, an image sensor, and a processing circuit. The light source is used to transmit a light signal toward the target object. Part of the light signal is reflected by the target object, forming an echo signal that is incident on the image sensor. The image sensor collects the echo signal and generates an electrical signal. The processing circuit processes this electrical signal to calculate the flight time of the light signal to and from the target object, thereby obtaining 3D information data of the target object. In some embodiments, the ToF camera can generate a depth image, which is processed by the processor to obtain point cloud data. The ToF camera can also directly generate point cloud data and send the point cloud data to the processor to generate a three-dimensional mesh model.

[0037] To achieve more consistent texture image mapping on a 3D mesh model and optimize the visual quality of the target object's 3D model, this embodiment of the present application provides a texture mapping method based on 3D facial reconstruction. Referring to Figure 2 , this method, provided in one embodiment of the present application, includes the following steps:

[0038] S201. Obtain a three-dimensional mesh model and a standard 3DMM model of the target object; wherein the standard 3DMM model is preset with a first area and 3D face key points.

[0039] In some embodiments, a method for obtaining a three-dimensional mesh model includes: obtaining point cloud data of a target object, meshing the point cloud data, and obtaining a three-dimensional mesh model. Please refer to Figures 3-4. Figure 3 is a schematic diagram of point cloud data provided by an embodiment of the present application. The point cloud data of the target object can be directly obtained by scanning the target object with a depth camera, or it can be indirectly obtained by performing coordinate system conversion after collecting a depth image with a depth camera. Figure 4 is a schematic diagram of a three-dimensional mesh model provided by an embodiment of the present application, showing that the three-dimensional mesh model is obtained by approximating the point cloud data using a triangular mesh. The three-dimensional mesh model can also be obtained by approximating the point cloud using a quadrangular mesh, which is not limited here.

[0040] In some embodiments, a method for obtaining a standard 3DMM model includes: constructing a face mesh model in a world coordinate system comprising a preset number of vertices, labeling and dividing the vertices of the face mesh model into regions, and obtaining a standard 3DMM model labeled with vertex numbers and having a preset first region. Referring to FIG5 , FIG5(a) is a schematic diagram of a face mesh model comprising a preset number of vertices provided in an embodiment of the present application. The face mesh model represents a face as a parameterized model, using a preset number of vertices to establish a mesh to describe the model's geometric shape. The model's geometric shape can be adjusted by adjusting the coordinate parameters of each vertex. FIG5(b) is a schematic diagram of a face mesh model labeled with vertex numbers provided in an embodiment of the present application. Each vertex in the face mesh model can be labeled with a number, or only some vertices can be labeled with a number. Vertices with the same number represent the same semantics. Furthermore, a processor can distinguish whether a vertex is a facial key point or a specific facial key point based on the vertex number. The facial key points are shown as circles in the figure. Figure 5 (c) is a schematic diagram of a face mesh model divided into different areas provided by an embodiment of the present application. The areas can be divided according to vertex numbers or facial key points or manually. A standard 3DMM model with a first area is preset. The first area includes at least one area, and the diagram shows the division of eye sub-area, nose sub-area, mouth sub-area and other sub-areas. Each vertex will only exist in one of the first areas.

[0041] S202: align the standard 3DMM model with the three-dimensional mesh model to obtain a registered 3DMM model.

[0042] The standard 3DMM model and the 3D mesh model have different vertex counts and geometric shapes. When registering the two models, the standard 3DMM model is used as the template model, and the 3D mesh model of the target object is used as the target model. Non-rigid deformation of the template model is performed to output a registered 3DMM model with the same geometry and number of vertices as the template model. In addition, the registered 3DMM model inherits the vertex numbering and region division information from the standard 3DMM model.

[0043] In some embodiments, registering the standard 3DMM model with the 3D mesh model to obtain a registered 3DMM model includes adjusting the coordinates of the vertices of the standard 3DMM model in a world coordinate system so that the geometric shape of the adjusted model approximates that of the 3D mesh model, thereby obtaining the registered 3DMM model. The adjusted model and the 3D mesh model are considered to have the same geometric shape if the geometric shapes are approximately the same within a tolerance range due to the difference in the number of vertices.

[0044] S203 , dividing the three-dimensional mesh model into regions according to the positional relationship between the vertices of the registered 3DMM model and the first region to obtain a second region of the three-dimensional mesh model.

[0045] The registered 3DMM model is deformed from the standard 3DMM model, so the first region will also deform adaptively. The inclusion relationship between the first region and the model's vertices will not change due to the model deformation. Since the ultimate goal is to perform texture mapping on the target object's 3D mesh model, a second region needs to be divided on the 3D mesh model.

[0046] In some embodiments, in order to more accurately divide the second area on the three-dimensional mesh model, the three-dimensional mesh model is divided into regions according to the positional relationship between the vertices of the aligned 3DMM model and the first region to obtain the second region of the three-dimensional mesh model, including: extracting a first vertex set from the aligned 3DMM model; matching the first vertex set with the vertices of the three-dimensional mesh model to obtain a second vertex set that has a mapping relationship with the first vertex set; dividing the second vertex set into regions according to the positional relationship between the first vertex set and the first region to obtain the second region of the three-dimensional mesh model.

[0047] It can be understood that the three-dimensional mesh model of the target object is constructed by collecting point cloud data (parallax image or depth image) of the target object through a depth camera. Its mesh topology is dense and has a large number of vertices. The 3DMM model after registration has fewer vertices than the three-dimensional mesh model of the target object. Each vertex in the 3DMM model after registration can find a corresponding or adjacent vertex in the three-dimensional mesh model of the target object. That is, the present application can determine the second area with more vertices in the three-dimensional mesh of the target object through the first area with fewer vertices in the 3DMM model after registration, which not only reduces the amount of calculation, but also helps to improve the accuracy of area division, thereby achieving efficient and high-precision texture mapping.

[0048] In some embodiments, the vertices numbered 1, 2, and 3 in the registered 3DMM model are located in the eye sub-region of the first region, the vertices numbered 4, 5, and 6 in the registered 3DMM model are located in the nose sub-region of the first region, and the vertices numbered 7, 8, and 9 in the registered 3DMM model are located in the mouth sub-region of the first region. Since the registered 3DMM model and the three-dimensional mesh model of the target object have the same shape, a second vertex set that forms a mapping relationship with the first vertex set numbered 1 to 9 can be identified in the three-dimensional mesh model of the target object, that is, according to the coordinate positions of the vertices numbered 1 to 9 in the first vertex set, the corresponding vertices are searched in the three-dimensional mesh model of the target object to obtain the second vertex set, and the second vertex set is further divided into regions based on the positional relationship between the first vertex set and the first region to obtain the eye sub-region of the second region, the nose sub-region of the second region, and the mouth sub-region of the second region.

[0049] In some embodiments, the first vertex set may be all vertices in the 3DMM model after registration, and vertices having a mapping relationship with all vertices in the 3DMM model after registration are searched in the three-dimensional mesh model of the target object to form a second vertex set. When the second vertex set is divided into regions, errors can be reduced and the accuracy of region division can be improved. In other embodiments, the first vertex set may be some vertices in the 3DMM model after registration, especially vertices located at the edge of the first region. Compared with selecting all vertices in the 3DMM model after registration as the first vertex set, selecting vertices located at the edge of the first region as the first vertex set can greatly reduce the amount of data running and improve computing efficiency.

[0050] In some embodiments, the first vertex set can also be all vertices in the first area of ​​the 3DMM model after registration (such as only selecting the eye sub-area), and vertices with a mapping relationship with all vertices in the first area of ​​the 3DMM model after registration are searched in the three-dimensional mesh model of the target object to form a second vertex set. Compared with selecting all vertices in the 3DMM model after registration as the first vertex set to obtain the second vertex set, it can reduce the amount of data running while improving the accuracy of area division.

[0051] In some embodiments, the first vertex numbered 1 in the 3DMM model after registration is located in the eye sub-region of the first area, and a second vertex that forms a mapping relationship with the first vertex numbered 1 is searched in the three-dimensional mesh model of the target object, and the second vertex will be divided into the eye sub-region of the second area.

[0052] In some embodiments, matching a first vertex set with the vertices of a three-dimensional mesh model to obtain a second vertex set that forms a mapping relationship with the first vertex set includes: using a nearest neighbor search method to match the first vertex set of the registered 3DMM model with the vertices of the three-dimensional mesh model to obtain a second vertex set that forms a mapping relationship with the first vertex set. For example, a vertex i is extracted from the first vertex set, and a nearest neighbor search method is used to filter out the vertex o closest to vertex i from several vertices of the three-dimensional mesh model based on the Euclidean distance. Vertex i and vertex o are then matched, and then other vertices are extracted from the first vertex set in turn, and then the vertex closest to it is filtered out from several vertices of the three-dimensional mesh model based on the Euclidean distance, and finally the matching is completed. Since it is necessary to filter out the second vertex set that forms a mapping relationship from a large number of vertices of the three-dimensional mesh model, the amount of data is huge and the amount of noise data is large. Using nearest neighbor search has high accuracy and high tolerance to outliers and noise.

[0053] S204: Acquire a texture image of the target object, and perform facial key point detection on the texture image to obtain 2D facial key points.

[0054] Facial key point detection involves locating key facial areas, including eyebrows, eyes, nose, mouth, and facial contours, within a given facial image. Depending on the accuracy of key point detection, key points can be 58, 81, or 106 points. Key points of varying numbers are annotated in facial images, resulting in different coordinates for each key point.

[0055] S205: Match the 2D facial key points with the 3D facial key points of the registered 3DMM model to obtain the head posture information of the texture image.

[0056] The texture image of the target object is used to map the 3D mesh model. Existing mapping methods are prone to texture misalignment due to factors such as pose errors during reconstruction, camera distortion, and synchronization between the texture camera and the depth camera. This can result in unsightly 3D facial models. Therefore, texture images are tested to determine if they meet the mapping requirements.

[0057] In some embodiments, some vertices of the registered 3DMM model are 3D facial keypoints, and the number of 3D facial keypoints is the same as the number of 2D facial keypoints. The 3D facial keypoints are located in the world coordinate system, while the 2D facial keypoints are located in the image coordinate system of the texture camera. Points in the 3D facial keypoints correspond one-to-one with points in the 2D facial keypoints. See Figure 6, which shows five 3D facial keypoints in the registered 3DMM model and five 2D facial keypoints in the texture image, with curves used to represent the correspondence between the keypoints.

[0058] In some embodiments, matching 2D facial key points with 3D facial key points in a registered 3DMM model to obtain head pose information for the texture image includes: calculating the projection relationship between the 2D facial key points and the 3D facial key points based on a first coordinate set of the 2D facial key points in the texture camera's image coordinate system and a second coordinate set of the 3D facial key points in the world coordinate system, the projection relationship including a rotation matrix R and a translation matrix T; and obtaining the head pose information for the texture image based on the projection relationship. See Figure 7 for a schematic diagram of head pose information, which includes yaw, roll, and pitch angles.

[0059] Specifically, the PnP (Pespective-n-Point) algorithm is used to solve the head posture information of the texture image, including: obtaining the first coordinate set (p1, p2...pn) of the 2D face key points in the image coordinate system of the texture camera, obtaining the second coordinate set (P1, P2...Pn) of the 3D face key points in the world coordinate system, and calculating the projection relationship between the 2D face key points and the 3D face key points; based on the projection relationship, the position relationship of the 2D face key points relative to the 3D face key points is calculated to obtain the head posture information of the texture image.

[0060] In some embodiments, the coordinates of some key points from the 2D facial key points (such as the outer corners of the left and right eyes, the tip of the nose, and the center of the mouth) are extracted to form a first coordinate set (p1, p2, p3, and p4). The coordinates of some key points from the 2D facial key points (the same key points as those extracted from the 2D facial key points, such as the outer corners of the left and right eyes, the tip of the nose, and the center of the mouth) are extracted to form a second coordinate set (p1, p2, p3, and p4). The projection relationship between these four pairs of key points is calculated to obtain head pose information. Selecting a smaller number of key points for head pose calculation can reduce the amount of data to run and improve computational efficiency. However, at least four pairs of key points must be selected to solve the rotation matrix R and translation matrix T to accurately determine the head pose information.

[0061] It should be noted that in addition to calculating the projection relationship between the 3D face key points in the aligned 3DMM model and the 2D face key points of the texture image, this application can also find the 3D face key points in the face mesh model of the target object based on the 3D face key points of the aligned 3DMM model, and then use the PnP algorithm to calculate the projection relationship between the 3D face key points in the face mesh model of the target object and the 2D face key points of the texture image. The calculation process is similar to the above process and will not be repeated here.

[0062] S206: Determine whether the head posture information meets the preset requirements. If so, map the texture image to the second area.

[0063] In some embodiments, mapping a three-dimensional mesh model of a target object requires capturing texture images from multiple different angles of the face and determining whether the texture image can be used for mapping based on the head pose information of the texture image. For example, the eye sub-region of the second area requires a texture image with head pose information that satisfies a first pose interval. The nose sub-region of the second area includes a left sub-region and a right sub-region, and requires a texture image with head pose information that satisfies a second pose interval and a third pose interval, respectively. The first pose interval is yaw∈[-5°, 5°], roll∈[-5°, 5°], and ptich∈[-5°, 5°]; the second pose interval is yaw∈[-40°, -30°], roll∈[-10°, 10°], and ptich∈[-10°, 10°]; and the third pose interval is yaw∈[30°, 40°], roll∈[-10°, 10°], and ptich∈[-10°, 10°].

[0064] In some embodiments, an eye sub-area of ​​the second area and a texture image of a posture interval corresponding to the area are selected, the vertices of the three-dimensional mesh model of the target object are projected onto the texture image, the mapping relationship between the vertices of the model and the pixels of the texture image are bound, and the mesh of the area is texture mapped.

[0065] In some embodiments, when the head pose information of the texture image is accurate, the projected 2D facial key points will be almost perfectly aligned with the 3D facial key points. However, when the result is inaccurate, the 3D facial key points and the 2D facial key points can be transformed into the same coordinate system based on the current rotation matrix R and translation vector t. The mean square error between the 3D facial key points and the 2D facial key points is calculated based on their coordinates to obtain the reprojection error. By iteratively changing the values ​​of the rotation matrix R and the translation vector t, the reprojection error is reduced, and a more accurate projection relationship and head pose information are obtained, so that the pose matching of the texture image is more accurate when mapping.

[0066] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0067] Corresponding to the texture mapping method based on 3D reconstruction of a human face proposed in the above embodiments, FIG8 shows a structural block diagram of the texture mapping method based on 3D reconstruction of a human face provided in the embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown. As shown in FIG8, the apparatus includes:

[0068] The model acquisition module 81 is used to acquire a three-dimensional mesh model and a standard 3DMM model of the target object; wherein the standard 3DMM model is preset with a first region and 3D facial key points;

[0069] The model registration module 82 is used to register the standard 3DMM model with the three-dimensional grid model to obtain a registered 3DMM model;

[0070] A region division module 83 is configured to divide the three-dimensional mesh model into regions according to the positional relationship between the vertices of the registered 3DMM model and the first region, thereby obtaining a second region of the three-dimensional mesh model;

[0071] A key point detection module 84 is used to obtain a texture image of the target object and perform facial key point detection on the texture image to obtain 2D facial key points;

[0072] The pose estimation module 85 is used to match the 2D face key points with the 3D face key points of the registered 3DMM model to obtain the head pose information of the texture image;

[0073] The mapping judgment module 86 is used to judge whether the head posture information meets the preset requirements. If so, the texture image is mapped to the second area.

[0074] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0075] Please refer to Figure 9, which is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. The electronic device in this embodiment includes a processor 92, a memory 93, and a computer program 94 stored in the memory 93 and executable by the processor 92, connected via a bus 91. When the processor 92 executes the computer program 94, it can implement the various steps of the above-mentioned embodiment of the texture mapping method based on 3D facial reconstruction.

[0076] The processor 92 of the electronic device is used to provide computing and control capabilities. The memory 93 of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program 94. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The electronic device may include, but is not limited to, a processor 92 and a memory 93. Those skilled in the art will appreciate that Figure 9 is merely an example of an electronic device and does not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown, or a combination of certain components, or different components. For example, the electronic device may also include a network interface, a display, and an input device connected via a system bus. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements a texture mapping method based on three-dimensional facial reconstruction. The display of the electronic device may be a liquid crystal display or an electronic ink display. The input device of the electronic device may be a touch layer covering the display, or keys, a trackball, or a touchpad provided on the electronic device housing, or an external keyboard, touchpad, or mouse.

[0077] The processor 92 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0078] In some embodiments, memory 93 may be an internal storage unit of the electronic device, such as a hard drive or memory. In other embodiments, memory 93 may also be an external storage device of the electronic device, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, memory 93 may include both an internal storage unit of the electronic device and an external storage device. Memory 93 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. Memory 93 may also be used to temporarily store data that has been output or is about to be output.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into one processor, or the functional modules can be separated into different processors, or two or more modules can be integrated into one processor. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned devices and equipment can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0080] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0081] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0083] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0084] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A texture mapping method based on three-dimensional reconstruction of human face, characterized in that: include: Acquire a three-dimensional mesh model and a standard 3DMM model of the target object; wherein the standard 3DMM model is preset with a first area and 3D face key points; Aligning the standard 3DMM model with the three-dimensional mesh model to obtain an aligned 3DMM model; According to the positional relationship between the vertices of the registered 3DMM model and the first area, the three-dimensional mesh model is divided into regions to obtain a second region of the three-dimensional mesh model; Acquire a texture image of the target object, and perform facial key point detection on the texture image to obtain 2D facial key points; Matching the 2D face key points with the 3D face key points of the registered 3DMM model to obtain the head posture information of the texture image; Determine whether the head posture information meets a preset requirement, and if so, map the texture image to the second area.

2. The texture mapping method based on three-dimensional reconstruction of human face according to claim 1, characterized in that: The method for obtaining the three-dimensional grid model comprises: Acquire point cloud data of the target object; The point cloud data is meshed to obtain a three-dimensional mesh model.

3. The texture mapping method based on three-dimensional reconstruction of human face according to claim 1, characterized in that: The method for obtaining the standard 3DMM model includes: Constructing a face mesh model with a preset number of vertices in the world coordinate system; The vertices of the face mesh model are labeled with numbers and divided into regions to obtain a standard 3DMM model labeled with vertex numbers and preset with a first region.

4. The texture mapping method based on three-dimensional reconstruction of human face according to claim 1, characterized in that: The step of registering the standard 3DMM model with the three-dimensional mesh model to obtain a registered 3DMM model comprises: The coordinates of the vertices of the standard 3DMM model in the world coordinate system are adjusted so that the adjusted model has the same geometric shape as the three-dimensional mesh model, thereby obtaining the registered 3DMM model.

5. The texture mapping method based on three-dimensional reconstruction of human face according to claim 1, characterized in that: The step of dividing the three-dimensional mesh model into regions based on the positional relationship between the vertices of the registered 3DMM model and the first region to obtain the second region of the three-dimensional mesh model comprises: Extracting a first vertex set from the registered 3DMM model; Matching the first vertex set with the vertices of the three-dimensional mesh model to obtain a second vertex set that forms a mapping relationship with the first vertex set; According to the positional relationship between the first vertex set and the first area, the second vertex set is divided into areas to obtain a second area of ​​the three-dimensional mesh model.

6. The texture mapping method based on three-dimensional reconstruction of human face according to claim 1, characterized in that: The step of matching the 2D face key points with the 3D face key points of the registered 3DMM model to obtain the head posture information of the texture image includes: Calculating a projection relationship between the 2D face key points and the 3D face key points according to a first coordinate set of the 2D face key points in the texture camera coordinate system and a second coordinate set of the 3D face key points in the world coordinate system; According to the projection relationship, head posture information of the texture image is obtained.

7. A texture mapping device based on three-dimensional reconstruction of human face, characterized in that: include: A model acquisition module, used to acquire a three-dimensional mesh model and a standard 3DMM model of a target object; wherein the standard 3DMM model is preset with a first area and 3D face key points; A model registration module, used for registering the standard 3DMM model with the three-dimensional mesh model to obtain a registered 3DMM model; A region division module, used for dividing the three-dimensional mesh model into regions according to the positional relationship between the vertices of the registered 3DMM model and the first region, to obtain a second region of the three-dimensional mesh model; A key point detection module is used to obtain a texture image of the target object and perform facial key point detection on the texture image to obtain 2D facial key points; A posture estimation module is used to match the 2D face key points with the 3D face key points of the registered 3DMM model to obtain the head posture information of the texture image; A mapping judgment module is used to judge whether the head posture information meets the preset requirements, and if so, map the texture image to the second area.

8. A 3D face reconstruction device, characterized in that: comprising a camera, a processor, and a memory storing a computer program executable by the processor; The camera is used to collect 3D information data and texture images of the target object; The processor is used to receive the 3D information data and generate a three-dimensional mesh model according to the 3D information data; The processor is also used to implement the steps of the method described in any one of claims 1 to 6 when executing the computer program.

9. An electronic device, characterized in that: The method comprises a processor and a memory storing a computer program; when the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images