Three-dimensional head model reconstruction method and device and electronic equipment

Abstract

The invention relates to the field of computer vision and graphics, in particular to a three-dimensional head model reconstruction method and device and electronic equipment. The method comprises the following steps: acquiring a plurality of head images of the same head, wherein the plurality of head images at least comprise a front face image and a side face image; respectively converting the plurality of head images into images of a Tensor object; acquiring face region images according to the image of the Tensor object, wherein each face region image at least comprises eye image information, nose image information and mouth image information; acquiring face key point data according to the face region image; and establishing a three-dimensional head model according to the face key point data. According to the embodiment of the invention, the combination of a traditional image processing method and a depth algorithm is realized, and the image data is processed by using a depth detection algorithm through a large amount of image data, so that the generated three-dimensional head model is close to a real object, the problems in the prior art are solved, and the accuracy and efficiency of reconstruction of the three-dimensional head model are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and graphics, and more specifically, to a three-dimensional head model reconstruction method and device, and electronic equipment. Background Art

[0002] 3D head model reconstruction is a basic and important technology, which is widely used in many scenarios such as facial animation, security authentication, augmented reality (AR), etc. The commonly used 3D head model reconstruction method is based on facial images for calculation, but the selection of single-view image data usually cannot provide enough details, has certain limitations, and is prone to data loss or errors, affecting the integrity and accuracy of the reconstruction results. At the same time, single-view image data is difficult to capture complex facial expressions, especially extreme expressions (such as laughing, frowning, etc.), and the accuracy and consistency of the reconstruction results are difficult to guarantee. These shortcomings limit its effectiveness in practical applications. Summary of the invention

[0003] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a three-dimensional head model reconstruction method and device, and electronic equipment for achieving the effect of fully modeling the head and thus obtaining an accurate three-dimensional head model.

[0004] According to a first aspect of the present application, a three-dimensional head model reconstruction method is provided, comprising:

[0005] Acquire a plurality of head images of the same head, wherein the plurality of head images at least include a frontal face image and a side face image;

[0006] Convert the plurality of head images into images of Tensor objects respectively;

[0007] Acquire a facial region image according to the image of the Tensor object, wherein each facial region image includes at least eye image information, nose image information, and mouth image information;

[0008] Acquiring facial key point data according to the facial region image;

[0009] A three-dimensional head model is established according to the facial key point data.

[0010] The three-dimensional head model reconstruction method can establish a head model through multiple head images of different viewing angles including frontal face images, side face images, etc., obtain a facial region image from the multiple head images, obtain facial key point data using the facial region image, and then perform modeling based on the facial key point data. The head model established according to the reconstruction method has a high degree of restoration and high accuracy, can display the original details of the head of the captured image, has a small occlusion area, and improves the feasibility of restoring a large area model.

[0011] Optionally, acquiring a facial region image according to the image of the Tensor object includes:

[0012] The image of the Tensor object is input into the face detection network to obtain the face area image.

[0013] The face detection network is trained with a large-scale data set to learn a more accurate representation of facial features, can accurately detect multiple acquired head images, has strong robustness to changes in lighting, expression, posture, etc., can identify and locate facial areas with high precision, and extract facial areas after identification and location to obtain facial area images. The face detection network is highly applicable in a variety of application scenarios, such as security monitoring, identity authentication, smart retail, etc., and its applicability can be improved by increasing its depth and width.

[0014] Optionally, the step of inputting the image of the Tensor object into a face detection network to obtain a face area image specifically includes:

[0015] Presetting a detection frame in the face detection network, wherein the detection target of the detection frame is a face area;

[0016] The face detection network and the detection frame are used to detect the image of the Tensor object to obtain a face area image.

[0017] The detection frame is a necessary tool for the face detection network to detect the face area. By setting the corresponding detection frame, the face area detection is performed on different head images obtained, which can significantly improve the detection speed while ensuring the detection accuracy, allowing the face detection network to quickly eliminate a large number of non-face areas in the early stage, reducing the amount of calculation for subsequent processing. The multi-level network gradually optimizes the results of face detection, making the final output face candidate frame more accurate. When dealing with complex backgrounds and face occlusion, the detection frame can show stronger robustness and can also adapt to head images of different sizes and resolutions. By using various data enhancement techniques, such as random cropping, rotation and scaling, the diversity and richness of training data are increased, which helps to improve the generalization ability and robustness of the face detection network. The detection frame can perform fast, accurate and robust detection tasks.

[0018] Optionally, the detecting the image of the Tensor object using the face detection network and the detection frame to obtain a face area image specifically includes:

[0019] Inputting the image of the Tensor object into the face detection network, and projecting the detection frame onto the input image of the Tensor object, to obtain a detection frame having a certain overlapping area in the image of the Tensor object;

[0020] For the detection frames with a certain overlapping area, a retained detection frame is retained;

[0021] Reprojecting the retained detection boxes into the input image of the Tensor object to obtain the area and confidence of each retained detection box;

[0022] Screening is performed according to the area and confidence of each retained detection frame to obtain a screened detection frame;

[0023] The filtered detection frame is used to extract the image region of the input image of the Tensor object to obtain a facial region image.

[0024] The detection frames with a certain overlapping area in the image of the Tensor object are obtained, and one detection frame is retained for the detection frames with a certain overlapping area. The purpose of the operation is to reduce the overlapping detection frames so as to retain the best detection frame, which can improve the detection accuracy of the detection frame and optimize the computing resources, and improve the performance of the target detection model; the screening process uses the confidence and area as the screening conditions, and can remove the detection frames with too small facial area and low confidence. Through the screening process, an effective detection frame can be obtained, so that the facial data can be obtained. A clearer and more accurate image is obtained, which reduces a certain amount of calculation, improves the efficiency of obtaining facial key point data, and is conducive to establishing a three-dimensional head model with high accuracy.

[0025] Optionally, acquiring facial key point data according to the facial region image includes:

[0026] The facial region image is input into a facial key point detection network to obtain corresponding facial key point data.

[0027] The facial key point data includes the contour points of the eyes, nose, mouth and other parts. These contour points are the basis for 3D face reconstruction. By accurately locating these contour points and combining them with a 3D Morphable Model (3DMM), the precise shape and position of the face can be reconstructed in 3D space. The reconstructed 3D face model can also be registered with the 2D face image using the facial key point data, so that texture mapping can be performed more accurately, thereby generating a 3D head model with real texture.

[0028] Optionally, the three-dimensional head model reconstruction method further includes:

[0029] Performing a second confidence calculation on the facial key point data to obtain a corresponding second confidence value;

[0030] Determine whether the second confidence value is less than a preset second confidence threshold, if so, remove the corresponding facial key point data, if not, retain the corresponding facial key point data;

[0031] The step of establishing a three-dimensional head model according to the facial key point data specifically includes:

[0032] A three-dimensional head model is established based on the retained facial key point data.

[0033] Making predictions based on the results of the second confidence calculation helps to improve the robustness of the algorithm, especially when facing difficult situations such as occlusion and lighting changes. The second confidence is used to evaluate the reliability of the three-dimensional Gaussian center and adjust the prediction of the Gaussian parameters accordingly, thereby improving the quality of three-dimensional reconstruction and new view synthesis.

[0034] Optionally, establishing a three-dimensional head model according to the facial key point data includes:

[0035] Fusing all the facial key point data to obtain a fusion result;

[0036] A three-dimensional head model is established according to the fusion result.

[0037] The fusion result usually refers to the integration of multiple key points (for example, key points in human posture estimation, feature points in facial recognition, etc.) into a unified data structure in the fields of computer vision, machine learning, image processing, etc., such as fusing the facial key point data into a matrix. The advantage of fusing into a matrix is ​​that it can easily integrate information from different sources and different types of data, which is convenient for unified processing and analysis; it is also that the data structure in the form of a matrix is ​​suitable for large-scale mathematical operations, and the method of linear algebra can be used to improve the calculation efficiency. At the same time, through matrix operations, the relationship and features between key points can be easily extracted, which is particularly important for pattern recognition and machine learning algorithms. Fusion of facial key point data into a matrix is ​​a powerful data processing technology, which has a wide range of applications in many fields, and with the development of technology, its advantages and application scope are still expanding.

[0038] Optionally, establishing a three-dimensional head model according to the fusion result includes:

[0039] Obtaining shape parameters, expression parameters, and position parameters according to the fusion result;

[0040] Optimizing the shape parameters, expression parameters, and position parameters to obtain an optimization result;

[0041] A three-dimensional head model is established according to the optimization result.

[0042] The optimization results include the results obtained by optimizing the shape parameters, expression parameters, and position parameters. By optimizing the parameters, the influence of factors such as system parameter errors and image extraction errors on the accuracy of the three-dimensional reconstruction model can be reduced, which can help the model better adapt to noise and outliers and improve the robustness of the three-dimensional reconstruction model. In the process of reconstructing the model, deep learning is often used in combination with reinforcement learning methods. Optimizing parameters can help optimize data collection, path planning, reconstruction algorithms, and result optimization. At the same time, by optimizing parameters, iterative steps can be reduced and convergence speed can be accelerated. The optimization results play a vital role in three-dimensional reconstruction. It can not only improve the accuracy and robustness of reconstruction, but also improve the efficiency and adaptability of the model, as well as optimize the geometry and motion parameters of the model, thereby obtaining a more accurate and practical three-dimensional model.

[0043] According to a second aspect of the present application, a three-dimensional head model reconstruction device is provided, comprising:

[0044] A first acquisition module, used to acquire a plurality of head images of the same head, wherein the plurality of head images at least include a frontal face image and a side face image;

[0045] A conversion module, used to convert the plurality of head images into images of Tensor objects respectively;

[0046] A second acquisition module, used to acquire a facial region image according to the image of the Tensor object, wherein each facial region image at least includes eye image information, nose image information, and mouth image information;

[0047] The third acquisition module is used to acquire facial key point data according to the facial area image.

[0048] A building module is used to build a three-dimensional head model according to the facial key point data.

[0049] The three-dimensional head model reconstruction device can realize three-dimensional head model reconstruction, the first acquisition module acquires a plurality of head images of the same head, the plurality of head images at least include a front face image and a side face image; the plurality of head images pass through a conversion module, the conversion module converts the plurality of head images into images of Tensor objects respectively; the images converted into Tensor objects pass through the second acquisition module, the second acquisition module acquires facial area images according to the images of the Tensor objects, each facial area image at least includes eye image information, nose image information, and mouth image information; the facial area images pass through a third acquisition module, the third module acquires facial key point data according to the facial area images; the facial key point data arrives at an establishment module, the establishment module establishes a three-dimensional head model according to the facial key point data.

[0050] According to a third aspect of the present application, an electronic device is provided, including:

[0051] a memory for storing one or more computer programs;

[0052] The processor implements the three-dimensional head model reconstruction method described in the first aspect when the one or more computer programs are executed by the processor.

[0053] The electronic device provides a complete system for reconstructing the three-dimensional head model. The memory stores images, the processor processes the stored images, and the processed images are then stored in the memory.

[0054] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a three-dimensional head model reconstruction method described in the first aspect when executed.

[0055] Based on any of the above aspects, a three-dimensional head model reconstruction method and device, electronic device and computer storage medium provided in the embodiments of the present application can realize the reconstruction of the three-dimensional head model, and use the data information between multiple images to improve the reconstruction accuracy. By observing the same object from different angles, the characteristics of the object can be captured more comprehensively, thereby improving the accuracy of reconstruction. Multi-view information provides detailed information of different parts of the object, which can better handle problems such as occlusion and lighting changes, and improve the robustness of reconstruction. The three-dimensional head model reconstruction method can provide more accurate, robust and high-quality three-dimensional reconstruction results by combining information from multiple perspectives, and is suitable for a variety of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A schematic application scenario diagram of a three-dimensional head model reconstruction method provided in this embodiment.

[0058] Figure 2 A schematic flow chart of a three-dimensional head model reconstruction method provided in this embodiment.

[0059] Figure 3 A schematic diagram of functional modules of a three-dimensional head model reconstruction device provided in this embodiment.

[0060] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this embodiment. DETAILED DESCRIPTION

[0061] The drawings of this application are for illustrative purposes only and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] The current 3D head model reconstruction is based on single-view images. Single-view data can only capture one angle of the face and it is difficult to fully reflect the full picture of the face. Especially for the side face, nose, eyebrows, cheekbones and other parts, single-view data usually cannot provide enough details. When single-view data encounters occlusion (such as jewelry), it is easy to cause data loss or errors, affecting the integrity and accuracy of the reconstruction results. Single-view reconstruction is sensitive to changes in facial posture. The reconstruction results under different postures may vary greatly, resulting in inconsistent reconstruction effects. Single-view data is difficult to capture complex expression changes, especially extreme expressions (such as laughing, frowning, etc.). These defects limit the application effect of single-view data in 3D reconstruction, making the reconstruction model results may not be accurate and reliable enough.

[0065] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0066] For example, the following is a schematic diagram of an application scenario of a three-dimensional head model reconstruction method provided in an embodiment of the present application. Figure 1 As shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100, the server 100 has an image processing function; the terminal device 200 has a streaming media playback function and may also have an image processing function.

[0067] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a vehicle-mounted terminal, etc., but is not limited thereto.

[0068] In an practicable manner, the server 100 and the terminal 200 may respectively execute the three-dimensional head model reconstruction method provided in the embodiment of the present application, or, optionally, the three-dimensional head model reconstruction method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0069] like Figure 2 As shown, this embodiment provides a three-dimensional head model reconstruction method, which may include the following steps:

[0070] S110, acquiring a plurality of head images of the same head, wherein the plurality of head images at least include a frontal face image and a side face image;

[0071] In this embodiment, the head can be a human head or an animal head; the multiple head images include head images of the same head at multiple angles, and the head images of the multiple perspectives include at least a front face image and a side face image, and the side face image can include a left face image and a right face image.

[0072] In an optional implementation, a device is used to capture head images. In specific implementation, a camera can be used to capture multiple images around the target head. The multiple images may include a front face image, a left face image, a right face image, etc., which can fully reflect the characteristics of the target head. The specific form of obtaining the head image is not specifically limited.

[0073] S120, converting the plurality of head images into images of Tensor objects respectively;

[0074] In this embodiment, the Tensor object refers to a tensor, which is actually expressed as a multidimensional array. The several head images are converted into images of the Tensor object respectively to facilitate subsequent image processing.

[0075] In an optional implementation, before converting the several head images into images of Tensor objects respectively, the several head images can also be normalized, that is, the pixel values ​​of the several head images are normalized from [0, 255] to [0, 1]. After the normalization processing is completed, the several head images are converted into images of Tensor objects. The conversion can use the torchvision. conversion function of Py Torch to convert the several head images, and the conversion method is not limited to this.

[0076] It can be understood that converting the multiple head images into images of Tensor objects can represent data of any dimension, which is conducive to deep training and facilitates recognition processing.

[0077] S130, acquiring a facial region image according to the image of the Tensor object, wherein each facial region image includes at least eye image information, nose image information, and mouth image information;

[0078] In this embodiment, the facial area image is acquired according to the image of the Tensor object through a face detection network. The face detection network can be a network model such as MTCNN, P-Net, R-Net, O-Net, CMS-RCNN, etc. The network model is widely used in the field of face detection, has fast processing speed, high detection efficiency, and can efficiently process a large amount of image data. The device can also be used to directly collect the facial area, but the requirements for the device are relatively high, requiring clear and accurate image data.

[0079] In a specific implementation process, each of the facial region images may also include other facial image information such as eyebrows.

[0080] In an optional implementation, a face detection network is used to perform target detection on several head images to obtain a face area image, a detection frame is preset in the face detection network, and the detection target of the detection frame is the face area, the several head images are respectively converted into images of Tensor objects, and then the images of the Tensor objects are input into the face detection network for detection. In the process of performing target detection on the several head images through the detection frame, it is necessary to perform overlap removal processing on the detection frame. Specifically, the image of the Tensor object is input into the face detection network, and the detection frame is projected onto the input image of the Tensor object to obtain a detection frame with a certain overlapping area in the image of the Tensor object; for the detection frames with a certain overlapping area, one retained detection frame is retained. Preferably, the detection frame can be subjected to non-maximum suppression processing, that is, the detection frames with more overlapping areas are eliminated. This is because during the target detection process, the target detection model may generate multiple detection frames for the same target, especially when the boundary of the target is not obvious or the target is partially blocked, which will cause overlap. Only the detection frame that is most likely to contain the target object is retained to avoid repeated detection, and the retained detection frame is re-projected onto the image of the Tensor object. In the specific implementation process, the judgment condition that the area overlaps more can be achieved by presetting the area overlap rate.

[0081] In a preferred embodiment, after the detection frames of the image projected onto the Tensor object are subjected to overlap removal processing, the detection frames can be further screened to improve the extraction effect of the facial area image. Specifically, the screening of the detection frames can include removing one or two of the detection frames whose corresponding facial area images are too small and whose confidence is low. Specifically, the retained detection frames are re-projected onto the input image of the Tensor object to obtain the area and confidence of each retained detection frame; the screened detection frames are screened according to the area and confidence of each retained detection frame; and the screened detection frames are used to extract the image area of ​​the input image of the Tensor object to obtain the facial area image. In specific implementation, an area threshold may be set for the area, and a first confidence threshold may be set for the confidence. By setting the area threshold, the area of ​​each detection frame is obtained, and then it is determined whether the area of ​​the facial region corresponding to each detection frame is less than the preset area threshold. If so, the corresponding detection frame is removed; otherwise, the corresponding detection frame is retained. For the first confidence threshold, the first confidence represents the prediction result of the facial region image corresponding to the detection frame. Only when the actual first confidence is higher than the preset first confidence threshold does it indicate that the facial region image corresponding to the obtained detection frame is valid. The first confidence calculation is performed on all detection frames to obtain the corresponding first confidence values. Then it is determined whether the first confidence value is less than the preset first confidence threshold. If so, the corresponding detection frame is removed; otherwise, the corresponding detection frame is retained.

[0082] Exemplarily, the further screening process includes removing detection frames with too small a face area, that is, for an image of the Tensor object with a size of 1920*1080, removing detection frames with an area less than 3000 pixels; and also includes removing detection frames with lower confidence levels, including removing detection frames with confidence levels lower than 0.5.

[0083] In a preferred implementation, since multiple candidate boxes appear when performing a target detection on the same head image, the largest detection box among the detection boxes after the screening process is selected to crop the image of the Tensor object, and the cropped facial area image is adjusted to a uniform size through a scaling operation, and the uniform size includes but is not limited to 224x224 pixels. The facial area image is obtained according to the operation method, and the operation method is not specifically limited.

[0084] It can be understood that the face detection network can be used to directly learn the detection results from the original image. The face area data obtained by the face detection network can provide very high detection accuracy, especially under complex backgrounds and different lighting conditions. The face detection network can detect faces of different sizes, which makes them applicable in a variety of scenarios. The method of obtaining the face area image includes but is not limited to this.

[0085] S140, acquiring facial key point data according to the facial region image;

[0086] In this embodiment, the method of acquiring facial key point data according to the facial region image may be through a facial key point detection network, or may be implemented through open source, or may be a data set annotation method.

[0087] In an optional implementation, the facial region image is input into the facial key point detection network to obtain facial key point data, and the obtained key point data also needs to be processed accordingly. The method of obtaining the facial key point data is not specifically limited.

[0088] In a preferred implementation, the processing includes performing a second confidence calculation on the facial key point data to obtain a corresponding second confidence value; judging whether the second confidence value is less than a preset second confidence threshold, if so, removing the corresponding facial key point data, otherwise retaining the corresponding facial key point data, the second confidence represents the prediction result of the acquired facial key point data, and only when the actual second confidence is higher than the preset second confidence threshold does it indicate that the acquired facial key point data is valid, specifically implemented by setting the second confidence and the corresponding second confidence threshold, first performing a second confidence calculation on the facial key point data to obtain a corresponding second confidence value, then judging whether the second confidence value is less than the preset second confidence threshold, if so, removing the corresponding facial key point data, otherwise retaining the corresponding facial key point data. Exemplarily, the second confidence threshold can be set to 0.6 or other values, and the key point data with high accuracy can be obtained by the data acquisition method, and the processing of the facial key point data includes but is not limited to this.

[0089] S150: Establish a three-dimensional head model according to the facial key point data.

[0090] In this embodiment, the method of establishing a three-dimensional head model based on the facial key point data includes a three-dimensional deformable model (3D Morphable Model, 3DMM), a deep learning-based method, a nonlinear 3DMM framework, a key point regression-based method, a PnP (Perspective-n-Point) algorithm, etc.

[0091] In an optional implementation, a three-dimensional head model is established according to the facial key point data, and the model is established using a three-dimensional deformable model (3D Morphable Model, 3DMM), specifically, the acquired facial key point data of multiple views are fused to obtain a fusion result, and then shape parameters, expression parameters, and position parameters are obtained according to the fusion result, the shape parameters, expression parameters, and position parameters are initialized to obtain an initialization result, and the three-dimensional head model is established according to the initialization result. In a specific implementation process, the fusion method can be to splice all facial key point data into a matrix as the fusion result, and then extract shape parameters, expression parameters and position parameters from the matrix, and initialize the shape parameters and expression parameters. The initialization of the position parameters is to initialize a camera 6DOF (6 Degrees of Freedom) parameter for each acquired view, which refers to six independent motion parameters of an object in a three-dimensional space. The six independent motion parameters include the movement of the object in the directions of the three coordinate axes of X, Y, and Z, which correspond to the forward / backward, left / right, up / down movement and the rotation of the object around the three coordinate axes of X, Y, and Z, which correspond to the roll angle (Roll, rotation around the X axis), pitch angle (Pitch, rotation around the Y axis) and yaw angle (Yaw, rotation around the Z axis), which can describe and control the position and direction of the object in the three-dimensional space, and establish a three-dimensional head model according to the shape parameters, expression parameters and 6DOF parameters. The method of establishing the model is not specifically limited.

[0092] In a preferred implementation, a back propagation algorithm is used to optimize rigid parameters and non-rigid parameters, wherein the rigid parameters include position parameters, which are specifically translation parameters, rotation parameters, and scaling parameters; the non-rigid parameters include shape parameters and expression parameters, etc., and a three-dimensional head model is restored based on the parameters optimized by the back propagation algorithm.

[0093] The technical solution for reconstructing a three-dimensional head model provided by the present application is described below in conjunction with a specific embodiment.

[0094] like Figure 3 As shown, the embodiment of the present application also provides a three-dimensional head model reconstruction device 210.

[0095] Optionally, the three-dimensional head model reconstruction device 210 may include a first acquisition module 211, a conversion module 212, a second acquisition module 213, a third acquisition module 214, and a building module 215, wherein:

[0096] A first acquisition module 211 is used to acquire a plurality of head images of the same head, wherein the plurality of head images at least include a front face image and a side face image;

[0097] In this embodiment, the first acquisition module 211 may be used to execute Figure 2 As shown in step S110, for a detailed description of the first acquisition module 211, reference may be made to the description of step S110.

[0098] A conversion module 212, used to convert the plurality of head images into images of Tensor objects respectively;

[0099] In this embodiment, the conversion module 212 can be used to perform Figure 2 As shown in step S120, for a detailed description of the conversion module 212, reference may be made to the description of step S120.

[0100] A second acquisition module 213, configured to acquire a facial region image according to the image of the Tensor object, wherein each facial region image at least includes eye image information, nose image information, and mouth image information;

[0101] In this embodiment, the second acquisition module 213 executes Figure 2 As shown in step S130, for a detailed description of the second acquisition module 213, reference may be made to the description of step S130.

[0102] The third acquisition module 214 is used to acquire facial key point data according to the facial region image.

[0103] In this embodiment, the third acquisition module 214 can be used to perform Figure 2 As shown in step S140, for a detailed description of the third acquisition module 214, reference may be made to the description of step S140.

[0104] The building module 215 is used to build a three-dimensional head model according to the facial key point data.

[0105] In this embodiment, the establishment module 215 can be used to execute Figure 2 As shown in step S150, for the specific description of the conversion module 211, reference may be made to the description of step S150.

[0106] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A three-dimensional head model reconstruction device provided in an embodiment of the present application can execute a three-dimensional head model reconstruction method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the three-dimensional head model reconstruction device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0107] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of a three-dimensional head model reconstruction method in combination with the embodiment of the present application can be directly embodied as a hardware encoding processor, or a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, and other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0108] The present application embodiment provides an electronic device 310, whose structure is as follows: Figure 4 The electronic device 310 may be Figure 1 The server 100 or the terminal 200 is shown.

[0109] like Figure 4 As shown, the electronic device 310 includes a memory 311, a processor 312, a communication module 313 and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313 and the input / output interface 314 can be connected and communicated through a bus 315.

[0110] The memory 311 is used to store one or more computer programs and transmit the code of the computer program to the processor 312; when the one or more computer programs are executed by the processor 312, the three-dimensional head model reconstruction method in the embodiment of the present application is implemented.

[0111] Optionally, the electronic device 310 can be connected to a network via a communication module 313 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 310 can be a digital computer in various forms, such as a desktop computer, a server, a workbench, a mainframe computer or other types of computers. The electronic device 310 can also be a mobile terminal in various forms, such as a smart phone, a tablet computer, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar mobile terminals.

[0112] Optionally, the electronic device 310 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 314. The electronic device 310 itself can have a display device, and can also be connected to other display devices through the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 314, so that data in the electronic device 310 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device 310, or it can be an external device connected to the electronic device 310 when needed.

[0113] Optionally, the memory 311 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0114] Optionally, the computer program stored in the processor 312 may be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 310.

[0115] Optionally, the processor 312 may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 312 executes various methods and processes of this embodiment, illustratively, such as a three-dimensional head model reconstruction method of an embodiment of the present application.

[0116] Optionally, the bus 315 may include a path for transmitting information. The bus 315 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 315 may be divided into an address bus, a data bus, a control bus, etc.

[0117] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 311 of the electronic device 310. When the computer program is executed by the processor 312, one or more steps of a three-dimensional head model reconstruction method of the embodiment of the present application can be performed.

[0118] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0119] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A three-dimensional head model reconstruction method, characterized in that: include: Acquire a plurality of head images of the same head, wherein the plurality of head images at least include a frontal face image and a side face image; Convert the plurality of head images into images of Tensor objects respectively; Acquire a facial region image according to the image of the Tensor object, wherein each facial region image includes at least eye image information, nose image information, and mouth image information; Acquiring facial key point data according to the facial region image; A three-dimensional head model is established according to the facial key point data.

2. A three-dimensional head model reconstruction method according to claim 1, characterized in that: The acquiring the facial region image according to the image of the Tensor object comprises: The image of the Tensor object is input into the face detection network to obtain the face area image.

3. A three-dimensional head model reconstruction method according to claim 2, characterized in that: Inputting the image of the Tensor object into a face detection network to obtain a face area image specifically includes: Presetting a detection frame in the face detection network, wherein the detection target of the detection frame is a face area; The face detection network and the detection frame are used to detect the image of the Tensor object to obtain the face area. Domain image.

4. A three-dimensional head model reconstruction method according to claim 3, characterized in that: The detecting the image of the Tensor object using the face detection network and the detection frame to obtain a face area image specifically includes: Inputting the image of the Tensor object into the face detection network, and projecting the detection frame onto the input image of the Tensor object, to obtain a detection frame having a certain overlapping area in the image of the Tensor object; For the detection frames with a certain overlapping area, a retained detection frame is retained; Reprojecting the retained detection boxes into the input image of the Tensor object to obtain the area and confidence of each retained detection box; Screening is performed according to the area and confidence of each retained detection frame to obtain a screened detection frame; The filtered detection frame is used to extract the image region of the input image of the Tensor object to obtain a facial region image.

5. A three-dimensional head model reconstruction method according to claim 1, characterized in that: The step of acquiring facial key point data according to the facial region image comprises: The facial region image is input into a facial key point detection network to obtain corresponding facial key point data.

6. A three-dimensional head model reconstruction method according to claim 1, characterized in that: Also includes: Performing a second confidence calculation on the facial key point data to obtain a corresponding second confidence value; Determine whether the second confidence value is less than a preset second confidence threshold, if so, remove the corresponding facial key point data, if not, retain the corresponding facial key point data; The step of establishing a three-dimensional head model according to the facial key point data specifically includes: A three-dimensional head model is established based on the retained facial key point data.

7. A three-dimensional head model reconstruction method according to any one of claims 1 to 6, characterized in that: The step of establishing a three-dimensional head model according to the facial key point data comprises: Fusing all the facial key point data to obtain a fusion result; A three-dimensional head model is established according to the fusion result.

8. A three-dimensional head model reconstruction method according to claim 7, characterized in that: The step of establishing a three-dimensional head model according to the fusion result comprises: Obtaining shape parameters, expression parameters, and position parameters according to the fusion result; Optimizing the shape parameters, expression parameters, and position parameters to obtain an optimization result; A three-dimensional head model is established according to the optimization result.

9. A three-dimensional head model reconstruction device, characterized in that: include: A first acquisition module, used to acquire a plurality of head images of the same head, wherein the plurality of head images at least include a frontal face image and a side face image; A conversion module, used to convert the plurality of head images into images of Tensor objects respectively; A second acquisition module, used to acquire a facial region image according to the image of the Tensor object, wherein each facial region image at least includes eye image information, nose image information, and mouth image information; The third acquisition module is used to acquire facial key point data according to the facial area image. A building module is used to build a three-dimensional head model according to the facial key point data.

10. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements a three-dimensional head model reconstruction method as described in any one of claims 1-8.

Images