Face model acquisition method and device, equipment and storage medium

By constructing and aligning the three-dimensional feature points of the facial scanning model, the digital human facial model is automatically acquired, which solves the inefficiency and accuracy problems caused by manual operation dependence in the existing technology, and achieves efficient and accurate facial model acquisition.

CN118941696APending Publication Date: 2024-11-12BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310519501.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The production of digital human facial models in the prior art relies on manual operations, resulting in inefficiency and difficulty in ensuring the accuracy of the model, especially when aligning the facial scanning model with the standard model through an image or depth scanner.

Method used

By obtaining the original image collection of multi-frames of the target face, a facial scanning model is constructed and two-dimensional feature points are extracted on the face, three-dimensional feature points are determined, and a reference facial model is generated based on these feature points, and then the preset standard model is automatically aligned to obtain the target face Model.

Benefits of technology

Automatic facial model acquisition is realized, which reduces the requirements for the technical level of production personnel, improves the accuracy and processing efficiency of facial models, and reduces the dependence of manual operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941696A_ABST
    Figure CN118941696A_ABST
Patent Text Reader

Abstract

The invention relates to a face model acquisition method, device and equipment and a storage medium, and the method comprises the steps: responding to an original image set of multiple frames of original images containing a target face, constructing a face scanning model based on the original image set, and extracting face two-dimensional feature points; determining a first three-dimensional feature point on the face scanning model based on the face two-dimensional feature point; generating a reference face model based on the face scanning model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model; and aligning the preset standard model with the reference face model to obtain a target face model. According to the method and the device, the face model can be automatically acquired, model alignment does not need to be performed depending on manual operation, the requirement on the technical level of manufacturing personnel can be reduced, the accuracy or precision of the acquired target face model can be ensured, a large amount of time and energy of the operating personnel can be saved, and the face model acquisition efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a facial model acquisition method, device, equipment and storage medium. Background Art

[0002] Digital human technology has prominent applications in CG (Computer Graphics) animation design, film production, and virtual idols. Among them, the production of digital human faces is an important part of digital human technology. At present, there are two main ways to produce digital human faces: one is to obtain a digital human face model from a standard model by manually pinching the face; the other is to align the face scan model obtained by an image or depth scanner with the standard face model, and then obtain a digital human face model based on the alignment result.

[0003] For the second method mentioned above, the relevant technology usually needs to rely on manual operation, which not only requires a high level of technical skills of the production personnel and is difficult to ensure the accuracy or precision of the facial model finally obtained, but also requires a lot of time and energy of the operator, which is inefficient. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the embodiments of the present disclosure provide a facial model acquisition method, apparatus, device and storage medium to solve the defects in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, a facial model acquisition method is provided, the method comprising:

[0006] In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points;

[0007] Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point;

[0008] Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model;

[0009] The preset standard model is aligned with the reference facial model to obtain a target facial model.

[0010] In some embodiments, constructing a facial scanning model based on the original image set includes:

[0011] The original image set is input into a preset three-dimensional reconstruction algorithm model to obtain the facial scanning model.

[0012] In some embodiments, extracting facial two-dimensional feature points based on the original image set includes:

[0013] Selecting at least two frames of target images from the original image set, wherein the orientation of the target face in the target images is a preset orientation;

[0014] Based on a preset deep learning model, feature points are extracted from the at least two frames of target images respectively to obtain the facial two-dimensional feature points.

[0015] In some embodiments, determining the first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point includes:

[0016] Determine a third three-dimensional feature point corresponding to the two-dimensional facial feature point based on a triangulation measurement method;

[0017] The third three-dimensional feature points are aligned to the facial scanning model to obtain the first three-dimensional feature points.

[0018] In some embodiments, the determining the third three-dimensional feature point corresponding to the two-dimensional facial feature point based on triangulation measurement method includes:

[0019] Normalizing the pixel coordinates of the two-dimensional facial feature points on the target image of each frame based on a predetermined camera internal parameter to obtain the normalized coordinates of the two-dimensional facial feature points corresponding to the target image of each frame;

[0020] The third three-dimensional feature point is determined based on the normalized coordinates and a predetermined camera pose corresponding to each frame of the target image.

[0021] In some embodiments, aligning the third three-dimensional feature points to the facial scan model to obtain the first three-dimensional feature points includes:

[0022] Determining the Euclidean distance between the third three-dimensional feature point and each point on the facial scan model;

[0023] The first three-dimensional feature point is determined based on the point on the facial scan model whose Euclidean distance meets the set condition.

[0024] In some embodiments, generating a reference facial model based on the facial scan model, the first three-dimensional feature points, and second three-dimensional feature points on a preset standard model includes:

[0025] Aligning the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result;

[0026] The first three-dimensional feature points on the facial scan model are adjusted based on the alignment result to obtain the reference facial model.

[0027] In some embodiments, the alignment result includes a scale factor between the facial scanning model and the preset standard model and a pose transformation matrix from the facial scanning model to the preset standard model;

[0028] The aligning the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result includes:

[0029] Constructing a first error function based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix;

[0030] The first error function is solved based on a preset optimization algorithm to obtain the scale factor and the posture transformation matrix.

[0031] In some embodiments, aligning the preset standard model with the reference facial model to obtain a target facial model includes:

[0032] Determining a model error between the preset standard model and the reference facial model;

[0033] The second three-dimensional feature points on the preset standard model are adjusted based on the model error to obtain the target facial model.

[0034] In some embodiments, determining a model error between the preset standard model and the reference facial model comprises:

[0035] Determine at least one morphological error between the reference facial model and the preset standard model, wherein the at least one morphological error comprises at least one of a chamfer distance error, a model side length error, and a model surface normal vector error;

[0036] constructing a second error function for characterizing the model error based on the at least one morphological error;

[0037] The step of adjusting the second three-dimensional feature points on the preset standard model based on the model error to obtain the target facial model includes:

[0038] Solving the second error function to obtain a function solution result;

[0039] The second three-dimensional feature points are adjusted based on the function solution result to obtain the target facial model.

[0040] According to a second aspect of an embodiment of the present disclosure, a facial model acquisition device is provided, the device comprising:

[0041] Constructing an extraction module, which is used to construct a face scanning model and extract two-dimensional facial feature points based on the original image set in response to obtaining a set of original images of multiple frames containing the target face;

[0042] A feature point determination module, configured to determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point;

[0043] A model generation module, used to generate a reference facial model based on the facial scanning model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model;

[0044] The model acquisition module is used to align the preset standard model with the reference facial model to obtain a target facial model.

[0045] In some embodiments, the constructing the extraction module comprises:

[0046] The model building unit is used to input the original image set into a preset three-dimensional reconstruction algorithm model to obtain the facial scanning model.

[0047] In some embodiments, the construction extraction module includes: a feature extraction unit;

[0048] The feature extraction unit is used for:

[0049] Selecting at least two frames of target images from the original image set, wherein the orientation of the target face in the target images is a preset orientation;

[0050] Based on a preset deep learning model, feature points are extracted from the at least two frames of target images respectively to obtain the facial two-dimensional feature points.

[0051] In some embodiments, the feature point determination module includes:

[0052] A second determining unit, configured to determine a third three-dimensional feature point corresponding to the two-dimensional facial feature point based on a triangulation method;

[0053] A feature point determination unit is used to align the third three-dimensional feature points onto the facial scanning model to obtain the first three-dimensional feature points.

[0054] In some embodiments, the second determining unit is further configured to:

[0055] Normalizing the pixel coordinates of the two-dimensional facial feature points on the target image of each frame based on a predetermined camera internal parameter to obtain the normalized coordinates of the two-dimensional facial feature points corresponding to the target image of each frame;

[0056] The third three-dimensional feature point is determined based on the normalized coordinates and a predetermined camera pose corresponding to each frame of the target image.

[0057] In some embodiments, the feature point determination unit is further configured to:

[0058] Determining the Euclidean distance between the third three-dimensional feature point and each point on the facial scan model;

[0059] The first three-dimensional feature point is determined based on the point on the facial scan model whose Euclidean distance meets the set condition.

[0060] In some embodiments, the model generation module includes:

[0061] an alignment unit, configured to align the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result;

[0062] A generating unit is used to adjust the first three-dimensional feature points on the facial scanning model based on the alignment result to obtain the reference facial model.

[0063] In some embodiments, the alignment result includes a scale factor between the facial scanning model and the preset standard model and a pose transformation matrix from the facial scanning model to the preset standard model;

[0064] The alignment unit is further configured to:

[0065] Constructing a first error function based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix;

[0066] The first error function is solved based on a preset optimization algorithm to obtain the scale factor and the posture transformation matrix.

[0067] In some embodiments, the model acquisition module includes:

[0068] an error determination unit, configured to determine a model error between the preset standard model and the reference facial model;

[0069] A model acquisition unit is used to adjust the second three-dimensional feature points on the preset standard model based on the model error to obtain the target facial model.

[0070] In some embodiments, the error determination unit is further configured to:

[0071] Determine at least one morphological error between the reference facial model and the preset standard model, wherein the at least one morphological error comprises at least one of a chamfer distance error, a model side length error, and a model surface normal vector error;

[0072] constructing a second error function for characterizing the model error based on the at least one morphological error;

[0073] The model acquisition unit is also used for:

[0074] Solving the second error function to obtain a function solution result;

[0075] The second three-dimensional feature points are adjusted based on the function solution result to obtain the target facial model.

[0076] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the device comprising:

[0077] a processor and a memory for storing a computer program;

[0078] Wherein, the processor is configured to implement, when executing the computer program:

[0079] In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points;

[0080] Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point;

[0081] Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model;

[0082] The preset standard model is aligned with the reference facial model to obtain a target facial model.

[0083] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the following is implemented:

[0084] In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points;

[0085] Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point;

[0086] Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model;

[0087] The preset standard model is aligned with the reference facial model to obtain a target facial model.

[0088] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0089] The present disclosure, in response to acquiring an original image set of multiple frames of original images containing a target face, constructs a facial scanning model based on the original image set and extracts facial two-dimensional feature points, determines first three-dimensional feature points on the facial scanning model based on the facial two-dimensional feature points, then generates a reference facial model based on the facial scanning model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model, and then aligns the preset standard model with the reference facial model to obtain a target facial model. This can achieve automated acquisition of facial models without relying on manual operation for model alignment, thereby reducing the technical level requirements for production personnel, thereby ensuring the accuracy or precision of the acquired target facial model, and saving a lot of time and energy of operators, thereby improving the processing efficiency of facial models.

[0090] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0092] Figure 1 is a flowchart of a facial model acquisition method according to an exemplary embodiment of the present disclosure;

[0093] Figure 2A is a flowchart showing how to extract facial two-dimensional feature points based on the original image set according to an exemplary embodiment of the present disclosure;

[0094] Figure 2B is a schematic diagram showing the extraction of facial two-dimensional feature points based on the original image set according to an exemplary embodiment of the present disclosure;

[0095] Figure 2C is a schematic diagram showing extraction of facial two-dimensional feature points based on the original image set according to another exemplary embodiment of the present disclosure;

[0096] Figure 3 is a flowchart showing how to determine the first three-dimensional feature points on the facial scan model based on the two-dimensional facial feature points according to an exemplary embodiment of the present disclosure;

[0097] Figure 4 is a flowchart showing how to determine the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on a triangulation measurement method according to an exemplary embodiment of the present disclosure;

[0098] Figure 5 is a flow chart showing how to obtain the first three-dimensional feature points according to an exemplary embodiment of the present disclosure;

[0099] Figure 6 is a flowchart showing how to generate a reference facial model according to an exemplary embodiment of the present disclosure;

[0100] Figure 7 is a flowchart showing how to align the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result according to an exemplary embodiment of the present disclosure;

[0101] Figure 8 is a flowchart showing how to align the preset standard model with the reference facial model to obtain a target facial model according to an exemplary embodiment of the present disclosure;

[0102] Fig.9A is a flowchart showing how to determine a model error between the preset standard model and the reference facial model according to an exemplary embodiment of the present disclosure;

[0103] Fig. 9B is a flowchart showing how to obtain the target facial model according to an exemplary embodiment of the present disclosure;

[0104] Fig. 9C is a schematic diagram of a facial scanning model according to an exemplary embodiment of the present disclosure;

[0105] Fig.9D is a schematic diagram of a preset standard model according to an exemplary embodiment of the present disclosure;

[0106] Fig.9E is a schematic diagram of a reference facial model according to an exemplary embodiment of the present disclosure;

[0107] Fig.9F is a schematic diagram of a target facial model according to an exemplary embodiment of the present disclosure;

[0108] Fig.10 is a block diagram of a facial model acquisition device according to an exemplary embodiment of the present disclosure;

[0109] Fig.11 is a block diagram of another facial model acquisition device according to an exemplary embodiment of the present disclosure;

[0110] Fig.12 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0111] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0112] At present, there are two main ways to produce digital human faces: one is to obtain a digital human face model from a standard model by manually pinching the face; the other is to align the face scanning model obtained by an image or depth scanner with the standard face model, and then obtain a digital human face model based on the alignment result.

[0113] Regarding the second method mentioned above, there are two solutions in the related technology: one is to provide a tool for 6 degrees of freedom (DoF) translation, rotation and scale, and align the two models by manually adjusting these parameters; the other is to select corresponding points between the facial scanning model and the standard facial model (such as corners of the eyes, eyebrows, corners of the mouth, ears, etc.), and align the two models based on the corresponding point position information.

[0114] However, both of the above solutions are too dependent on manual operation, which requires a lot of time and energy from the operator, is inefficient, and has low precision and is prone to errors. For example, if the scale setting in the first solution is wrong or the corresponding point selection in the second solution is wrong, the accuracy of the alignment of the two models will be affected, which will in turn affect the accuracy of the final facial model.

[0115] In view of this, the present disclosure provides the following facial model acquisition method, device, equipment and storage medium to solve the defects in the related art.

[0116] Figure 1 is a flowchart of a facial model acquisition method according to an exemplary embodiment; the method of this embodiment can be executed by a facial model acquisition device, which can be implemented by software and / or hardware and can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablet computers, etc.), wearable devices (such as glasses, watches, etc.), etc. Specifically, Figure 1 As shown, the method includes the following steps S101-S104:

[0117] In step S101, in response to acquiring an original image set of multiple frames of original images containing a target face, a facial scanning model is constructed based on the original image set and two-dimensional facial feature points are extracted;

[0118] In this embodiment, multiple frames of original images of the target face can be captured by an image acquisition device (such as a camera, etc.) to form an original image set, and then the electronic device can respond to the acquisition of the original image set, build a facial scanning model based on the set, and extract two-dimensional facial feature points.

[0119] For example, the image acquisition device may capture multiple frames of original images of the target face based on set shooting parameters, wherein the shooting parameters may include at least one of focal length, brightness, and exposure time.

[0120] In some implementations, the electronic device may input the original image set into a preset 3D reconstruction algorithm model to obtain the facial scan model. Exemplarily, the 3D reconstruction algorithm model may include algorithms from software such as Agisoft, RealityCaptrue, and Colmap, or algorithms in pre-written 3D reconstruction codes, which are not limited in this embodiment.

[0121] It is worth noting that the above-mentioned method of extracting facial two-dimensional feature points based on the original image set can refer to the image feature point extraction solution in the related art, and this embodiment does not limit this.

[0122] In another embodiment, the above-mentioned method of extracting facial two-dimensional feature points based on the original image set can also refer to the following Figure 2A The illustrated embodiment will not be described in detail here.

[0123] In step S102, first three-dimensional feature points on the facial scan model are determined based on the two-dimensional facial feature points.

[0124] In this embodiment, after the facial two-dimensional feature points are extracted, the first three-dimensional feature points on the facial scan model can be determined based on the facial two-dimensional feature points.

[0125] In some implementations, the coordinates of the same two-dimensional facial feature point corresponding to the three-dimensional space may be determined by combining the pixel coordinates of the same two-dimensional facial feature point on each frame of the original image, thereby obtaining the first three-dimensional feature point.

[0126] In another embodiment, the above method of determining the first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point can also refer to the following Figure 3 The illustrated embodiment will not be described in detail here.

[0127] In step S103, a reference facial model is generated based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model.

[0128] In this embodiment, after the facial scanning model is constructed and the first three-dimensional feature points are determined, a reference facial model is generated in combination with the second three-dimensional feature points on the preset standard model.

[0129] The preset standard model is a pre-built standard facial model of a digital human.

[0130] In some implementations, the first three-dimensional feature points on the facial scan model may be adjusted according to the second three-dimensional feature points, so that the adjusted facial scan model is used as the reference facial model.

[0131] It is worth noting that since the reference facial model is obtained by adjusting the facial scanning model, and the facial scanning model is obtained by three-dimensional reconstruction based on the original image set of the target face, the reference facial model can retain more details of the target face. Therefore, the reference facial model can be used as a fine alignment standard for the above-mentioned preset standard model in the future to obtain the target facial model.

[0132] In another embodiment, the above method of generating a reference face model can also refer to the following Figure 6 The illustrated embodiment will not be described in detail here.

[0133] In step S104, the preset standard model is aligned with the reference facial model to obtain a target facial model.

[0134] In this embodiment, after obtaining the above-mentioned reference facial model, the preset standard model can be aligned with the reference facial model to obtain a target facial model.

[0135] In some embodiments, aligning the preset standard model with the reference facial model may include adjusting the coordinates of a second three-dimensional feature point on the preset standard model according to the coordinates of the three-dimensional feature point on the reference facial model, thereby achieving a fine alignment of the preset standard model, so that the preset standard model after the fine alignment can be used as the target facial model.

[0136] In another embodiment, the above method of obtaining the target face model can also refer to the following Figure 8 The illustrated embodiment will not be described in detail here.

[0137] As can be seen from the above description, the method of this embodiment responds to obtaining an original image set of multiple frames of original images containing a target face, constructs a facial scanning model based on the original image set and extracts facial two-dimensional feature points, and determines the first three-dimensional feature points on the facial scanning model based on the facial two-dimensional feature points, and then generates a reference facial model based on the facial scanning model, the first three-dimensional feature points and the second three-dimensional feature points on a preset standard model, and then aligns the preset standard model with the reference facial model to obtain a target facial model. This can realize automated acquisition of facial models without relying on manual operation for model alignment, thereby reducing the technical level requirements for the production personnel, thereby ensuring the accuracy or precision of the acquired target facial model, and saving a lot of time and energy of the operator, thereby improving the processing efficiency of the facial model.

[0138] Figure 2A is a flowchart showing how to extract facial two-dimensional feature points based on the original image set according to an exemplary embodiment of the present disclosure; Figure 2B is a schematic diagram showing the extraction of facial two-dimensional feature points based on the original image set according to an exemplary embodiment of the present disclosure; Figure 2C is a schematic diagram showing the extraction of facial two-dimensional feature points based on the original image set according to yet another exemplary embodiment of the present disclosure.

[0139] This embodiment is based on the above embodiment and takes how to extract facial two-dimensional feature points based on the original image set as an example to illustrate.

[0140] like Figure 2A As shown, the extraction of facial two-dimensional feature points based on the original image set described in the above step S101 may include the following steps S201-S202:

[0141] In step S201, at least two frames of target images are selected from the original image set.

[0142] In this embodiment, when extracting two-dimensional facial feature points based on the original image set, at least two frames of target images may be selected from the original image set based on the orientation of the target face in the original image.

[0143] The orientation of the target face in the target image is a preset orientation (eg, front orientation, etc.).

[0144] In some embodiments, at least two frames of target images can be selected from the original image set based on the prior knowledge that the length ratio of the two eyes in the front view of the face is roughly the same. Exemplarily, the original images in the original image set whose length ratio of the two eyes is within a set ratio range (e.g., a range of 0.75 to 1.15, etc.) can be determined as target images. In order to ensure the accuracy and quality of the two-dimensional feature point extraction, the number of target images selected can be required to be more than two frames. Exemplarily, 52 two-dimensional feature points (i.e., 52 two-dimensional feature points) can be selected from each frame of the target image. Figure 2B to Figure 2C 0 to 51 in the target image shown in ). It is worth noting that the number of the two-dimensional feature points is only used for exemplary description. In specific applications, more or fewer feature points can be selected based on actual needs, and this embodiment does not limit this.

[0145] In step S202, feature points of the at least two frames of target images are extracted respectively based on a preset deep learning model to obtain the facial two-dimensional feature points.

[0146] In this embodiment, after selecting at least two frames of target images from the original image set, feature points of the at least two frames of target images can be extracted respectively based on a preset deep learning model to obtain the facial two-dimensional feature points.

[0147] Among them, the above-mentioned preset deep learning model can be selected based on actual business needs, such as being set to the MediaPipe framework in the related technology, the dlib library of Python, etc., which is not limited in this embodiment.

[0148] As can be seen from the above description, this embodiment selects at least two frames of target images from the original image set, and then extracts feature points of the at least two frames of target images based on a preset deep learning model to obtain the facial two-dimensional feature points. This can accurately extract the facial two-dimensional feature points from the at least two frames of target images selected from the original image set, and then ensure the accuracy of the first three-dimensional feature points on the facial scanning model that are subsequently determined based on the facial two-dimensional feature points. This can lay a reliable foundation for the subsequent generation of a reference facial model based on the first three-dimensional feature points and the acquisition of a target facial model based on the reference facial model, and can realize the automated acquisition of facial models, ensuring the accuracy or precision of the acquired target facial model, and improving the efficiency of facial model acquisition.

[0149] Figure 3 It is a flowchart showing how to determine the first three-dimensional feature points on the facial scanning model based on the two-dimensional facial feature points according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to determine the first three-dimensional feature points on the facial scanning model based on the two-dimensional facial feature points.

[0150] like Figure 3 As shown, the determination of the first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point in the above step S102 may include the following steps S301-S302:

[0151] In step S301, a third three-dimensional feature point corresponding to the two-dimensional facial feature point is determined based on a triangulation method.

[0152] In this embodiment, after the facial two-dimensional feature points are extracted based on the original image set, the third three-dimensional feature points corresponding to the facial two-dimensional feature points can be determined based on the triangulation method.

[0153] The specific explanation and description of the above triangulation method can be found in the records in the relevant technology, and this embodiment does not limit this.

[0154] In another embodiment, the above method of determining the first three-dimensional feature point on the facial scan model can also refer to the following Figure 4 The illustrated embodiment will not be described in detail here.

[0155] In step S302, the third three-dimensional feature points are aligned to the facial scanning model to obtain the first three-dimensional feature points.

[0156] In this embodiment, after determining the third three-dimensional feature point corresponding to the two-dimensional facial feature point, the third three-dimensional feature point can be aligned to the facial scanning model, so that the point on the facial scanning model that is aligned with the third three-dimensional feature point can be determined as the above-mentioned first three-dimensional feature point.

[0157] In another embodiment, the above method of aligning the third three-dimensional feature point to the facial scan model can also refer to the following Figure 5 The illustrated embodiment will not be described in detail here.

[0158] From the above description, it can be seen that this embodiment determines the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on the triangulation measurement method, and aligns the third three-dimensional feature points to the facial scanning model to obtain the first three-dimensional feature points. This can achieve accurate determination of the first three-dimensional feature points on the facial scanning model based on the two-dimensional facial feature points, and then subsequently generate a reference facial model based on the first three-dimensional feature points and the second three-dimensional feature points on the preset standard model, and obtain a target facial model based on the reference facial model. This can realize automated acquisition of facial models, thereby ensuring the accuracy or precision of the acquired target facial model and improving the efficiency of facial model acquisition.

[0159] Figure 4It is a flowchart showing how to determine the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on the triangulation method according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses the example of how to determine the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on the triangulation method to exemplify.

[0160] like Figure 4 As shown, the determination of the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on the triangulation method in the above step S301 may include the following steps S401-S402:

[0161] In step S401, the pixel coordinates of the facial two-dimensional feature points on the target image of each frame are normalized based on predetermined camera internal parameters to obtain the normalized coordinates of the facial two-dimensional feature points corresponding to the target image of each frame.

[0162] For example, suppose the pixel coordinates of the same facial 2D feature point on the target image of n frames are u = u1, u2, u…, u n , and the camera poses corresponding to the n-frame target images are T = T1, T2, T…, T n , then for the pixel coordinate u on the target image of the i-th (i=1, 2, ..., n) frame i (u i , v i ), it can be normalized by the following formula (4-1) to obtain the normalized coordinate u′ i (x, y, 1):

[0163]

[0164] In the above formula, c x 、c y 、f x 、f y These are the internal parameters of the camera, which can be set before the camera leaves the factory or obtained through calibration afterwards.

[0165] Specifically, c x 、c y are the horizontal (x-axis direction) and vertical y-axis direction offsets of the image origin relative to the optical center imaging point; f x =f / dx,f y =f / dy, where f is the focal length of the camera, dx and dy represent the length units occupied by a pixel in the x-axis direction and the y-axis direction respectively, that is, the size of the actual physical value represented by a pixel.

[0166] In some embodiments, the camera pose of the target image can be determined based on the 3D reconstruction process of the face scanning model in step S101, that is, the input of the 3D reconstruction algorithm model can be a set of original images, and the output can be the face scanning model and the camera pose of each frame of the original image. The target image is selected from the original image, so the camera pose of the target image can be known.

[0167] In step S402, the third three-dimensional feature point is determined based on the normalized coordinates and the predetermined camera pose corresponding to each frame of the target image.

[0168] In this embodiment, after the normalized coordinates of the facial two-dimensional feature points corresponding to the target image in each frame are obtained, the third three-dimensional feature points can be determined based on the normalized coordinates and the predetermined camera pose corresponding to the target image in each frame.

[0169] For example, assume that the camera pose corresponding to the n-frame target image is T = [R t], which is a 3x4 matrix, where R is the 3x3 rotation matrix from the world coordinate system to the camera coordinate system, and t is the 3x1 translation vector from the world coordinate system to the camera coordinate system. i (x, y, 1)=u′1, u′2, u′..., u′ n After that, the coordinates P (X, Y, Z) of the three-dimensional feature points corresponding to the two-dimensional facial feature points can be calculated by the following formula (4-2):

[0170]

[0171] In the above formula, is the cross product operation.

[0172] On this basis, by performing SVD (Singular Value Decomposition) operation on the above formula (4-2), the three-dimensional coordinates P (X, Y, Z), that is, the third three-dimensional feature point mentioned above, can be obtained.

[0173] In fact, each three-dimensional space point P can be calculated by the feature points of two frames of target images. The reason why multiple frames of target images are used for calculation in this embodiment is to improve the robustness of the calculation result and prevent abnormal values ​​from occurring.

[0174] As can be seen from the above description, this embodiment normalizes the pixel coordinates of the facial two-dimensional feature points on the target image of each frame based on the predetermined camera internal parameters, so as to obtain the normalized coordinates of the facial two-dimensional feature points corresponding to the target image of each frame, and then determines the third three-dimensional feature points based on the normalized coordinates and the predetermined camera pose corresponding to each frame of the target image, so as to accurately determine the third three-dimensional feature points corresponding to the facial two-dimensional feature points, thereby laying a foundation for the subsequent determination of the first three-dimensional feature points based on the third three-dimensional feature points, and then generating a reference facial model based on the first three-dimensional feature points and the second three-dimensional feature points on the preset standard model, and obtaining the target facial model based on the reference facial model, so as to realize the automatic acquisition of the facial model, and then ensure the accuracy or precision of the acquired target facial model, and improve the efficiency of facial model acquisition.

[0175] Figure 5 It is a flowchart showing how to obtain the first three-dimensional feature points according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment takes how to obtain the first three-dimensional feature points as an example for exemplary description.

[0176] like Figure 5 As shown, the above step S302 of aligning the third three-dimensional feature points to the facial scan model to obtain the first three-dimensional feature points may include the following steps S501-S502:

[0177] In step S501, the Euclidean distance between the third three-dimensional feature point and each point on the facial scan model is determined.

[0178] In this embodiment, after the third three-dimensional feature point corresponding to the two-dimensional facial feature point is determined, the Euclidean distance between the third three-dimensional feature point and each point on the facial scanning model can be determined respectively.

[0179] For example, suppose the third 3D feature point set is Ps, where the i-th point is Ps i , the Euclidean distance between this point and each point on the facial scan model can be calculated.

[0180] In step S502, the first three-dimensional feature point is determined based on the point on the facial scan model whose Euclidean distance meets the set condition.

[0181] In this embodiment, after determining the Euclidean distance between the third three-dimensional feature point and each point on the facial scanning model, the point on the facial scanning model whose Euclidean distance meets the set conditions (such as the shortest Euclidean distance, etc.) can be determined as the first three-dimensional feature point.

[0182] For example, the facial scan model and the i-th point Ps i The point with the shortest Euclidean distance between them is determined as the point Ps i The corresponding first three-dimensional feature points can thus obtain the first three-dimensional feature point set Qs.

[0183] From the above description, it can be seen that this embodiment determines the Euclidean distance between the third three-dimensional feature point and each point on the facial scanning model, and determines the first three-dimensional feature point based on the point on the facial scanning model where the Euclidean distance meets the set conditions. This can achieve accurate acquisition of the first three-dimensional feature point based on the third three-dimensional feature point and the facial scanning model, and further can achieve subsequent generation of a reference facial model based on the first three-dimensional feature point and the second three-dimensional feature point on a preset standard model, and obtain a target facial model based on the reference facial model, so as to achieve automated acquisition of the facial model, thereby ensuring the accuracy or precision of the acquired target facial model and improving the efficiency of facial model acquisition.

[0184] Figure 6 It is a flowchart showing how to generate a reference facial model according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment takes how to generate a reference facial model as an example for exemplary description.

[0185] like Figure 6 As shown, the generation of the reference facial model based on the facial scan model, the first three-dimensional feature points and the second three-dimensional feature points on the preset standard model described in the above step S103 may include the following steps S601-S602:

[0186] In step S601, the first three-dimensional feature point is aligned with the second three-dimensional feature point to obtain an alignment result.

[0187] In this embodiment, after the first three-dimensional feature point on the facial scanning model is determined, the first three-dimensional feature point can be aligned with to obtain an alignment result.

[0188] Among them, the above-mentioned alignment result may include the transformation parameters between the facial scanning model and the preset standard model, based on which the first three-dimensional feature points on the facial scanning model can be converted (limited by the calculation accuracy of the transformation parameters, this is actually an "approximate conversion" rather than a "precise conversion") into second three-dimensional feature points on the preset standard model.

[0189] Exemplarily, the transformation parameters may include a scale factor between the facial scanning model and the preset standard model and a posture transformation matrix from the facial scanning model to the preset standard model. On this basis, the above-mentioned method of aligning the first three-dimensional feature points with the second three-dimensional feature points can also refer to the following Figure 7 The illustrated embodiment will not be described in detail here.

[0190] In step S602, the first three-dimensional feature points on the facial scan model are adjusted based on the alignment result to obtain the reference facial model.

[0191] In this embodiment, after the first three-dimensional feature points are aligned with the second three-dimensional feature points and an alignment result is obtained, the first three-dimensional feature points on the facial scanning model can be adjusted based on the alignment result, so that the adjusted facial scanning model can be used as a reference facial model.

[0192] As can be seen from the above description, this embodiment aligns the first three-dimensional feature points with the second three-dimensional feature points to obtain an alignment result, and adjusts the first three-dimensional feature points on the facial scanning model based on the alignment result to obtain the reference facial model. This can accurately generate a reference facial model based on the facial scanning model, the first three-dimensional feature points and the second three-dimensional feature points on the preset standard model, so that the preset standard model can be subsequently aligned with the reference facial model to obtain a target facial model. This can improve the level of automation in acquiring facial models, reduce the reliance on manual operations in the model alignment process, thereby reducing the technical level requirements for the production personnel, ensuring the accuracy or precision of the acquired target facial model, saving the operator a lot of time and energy, and improving the efficiency of facial model acquisition.

[0193] Figure 7 It is a flowchart showing how to align the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to align the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result.

[0194] In this embodiment, the alignment result of the first three-dimensional feature point and the second three-dimensional feature point may include a scale factor between the facial scanning model and the preset standard model and a posture transformation matrix from the facial scanning model to the preset standard model.

[0195] On this basis, if Figure 7As shown, the alignment of the first three-dimensional feature point with the second three-dimensional feature point in the above step S601 to obtain an alignment result may include the following steps S701-S702:

[0196] In step S701, a first error function is constructed based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix.

[0197] In this embodiment, when the first three-dimensional feature points are aligned with the second three-dimensional feature points, a first error function may be constructed based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix.

[0198] For example, assuming that the first three-dimensional feature point set on the facial scan model is Qs, and the second three-dimensional feature point set on the preset standard model is Qs′, the following equation (7-1) can be established to solve the scale factor s and the pose transformation matrix T target The first error function C rough (s,T target ):

[0199]

[0200] In the above formula, n is the number of second three-dimensional feature points in the second three-dimensional feature point set.

[0201] In step S702, the first error function is solved based on a preset optimization algorithm to obtain the scaling factor and the posture transformation matrix.

[0202] In this embodiment, after constructing the first error function, the first error function can be solved based on a preset optimization algorithm to obtain the scaling factor and the posture transformation matrix.

[0203] In some embodiments, the above preset optimization algorithm can be used to solve the first error function C rough (s,T target ) to obtain the scale factor s and the pose transformation matrix T target .

[0204] It is worth noting that the above-mentioned preset optimization algorithm can be selected from relevant technologies based on actual business needs, such as selecting the Ceres optimization algorithm, etc., which is not limited in this embodiment.

[0205] On this basis, when the above scale factor s and posture transformation matrix T are obtained target After that, the i-th point Qs on the facial scanning model can be adjusted according to the following formula (7-2): i , get the adjusted i-th point

[0206]

[0207] Then, based on the adjusted i points The corresponding three-dimensional feature point set Qs u In other words, after adjusting each first three-dimensional feature point on the facial scan model according to the above formula (7-2), a reference facial model can be obtained. The set of three-dimensional feature points on the reference facial model is Qs u .

[0208] From the above description, it can be seen that this embodiment constructs a first error function based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor and the pose transformation matrix, and solves the first error function based on a preset optimization algorithm to obtain the scale factor and the pose transformation matrix. This can accurately calculate the alignment result of the first three-dimensional feature points and the second three-dimensional feature points, and then subsequently obtain the reference facial model based on the alignment result, which can lay the foundation for subsequently obtaining the target facial model according to the alignment result of the preset standard model and the reference facial model, and can improve the automation level of acquiring facial models, ensure the accuracy or precision of the acquired target facial model, and improve the efficiency of facial model acquisition.

[0209] Figure 8 It is a flowchart showing how to align the preset standard model with the reference facial model to obtain a target facial model according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to align the preset standard model with the reference facial model to obtain a target facial model.

[0210] like Figure 8 As shown, the above step S104 aligns the preset standard model with the reference facial model to obtain a target facial model, which may include the following steps S801-S802:

[0211] In step S801, a model error between the preset standard model and the reference facial model is determined.

[0212] In this embodiment, after the reference facial model is obtained, the model error between the preset standard model and the reference facial model may be determined.

[0213] In some embodiments, the method for determining the above-mentioned model error can refer to the method for determining the model error in the related art, and this embodiment does not limit this.

[0214] In other embodiments, the above-mentioned method for determining the model error can also refer to the following Fig.9A The illustrated embodiment will not be described in detail here.

[0215] In step S802, the second three-dimensional feature points on the preset standard model are adjusted based on the model error to obtain the target facial model.

[0216] In this embodiment, after the model error between the preset standard model and the reference facial model is determined, the second three-dimensional feature points on the preset standard model can be adjusted based on the model error to obtain the target facial model.

[0217] In some embodiments, after determining the above-mentioned model error, the second three-dimensional feature points on the preset standard model can be adjusted with the goal of reducing the numerical value of the model error, so that the target facial model can be determined based on the adjusted preset standard model.

[0218] In some other embodiments, the method for obtaining the target facial model can also refer to the following Fig. 9B The illustrated embodiment will not be described in detail here.

[0219] From the above description, it can be seen that this embodiment obtains the target facial model by determining the model error between the preset standard model and the reference facial model, and adjusting the second three-dimensional feature point on the preset standard model based on the model error. This can achieve accurate acquisition of the target facial model based on the alignment result of the preset standard model and the reference facial model, and can achieve automated acquisition of facial models without relying on manual operation for model alignment. This can reduce the technical level requirements for the production personnel, ensure the accuracy or precision of the acquired target facial model, save the operator a lot of time and energy, and improve the efficiency of facial model acquisition.

[0220] Fig.9A It is a flowchart showing how to determine the model error between the preset standard model and the reference facial model according to an exemplary embodiment of the present disclosure; based on the above-mentioned embodiment, this embodiment uses the example of how to determine the model error between the preset standard model and the reference facial model to exemplify.

[0221] like Fig.9A As shown, the determination of the model error between the preset standard model and the reference facial model in the above step S801 may include the following steps S901-S902:

[0222] In step S901, at least one morphological error between the reference facial model and the preset standard model is determined.

[0223] In this embodiment, after the reference facial model is determined, at least one morphological error between the reference facial model and the preset standard model may be determined.

[0224] Among them, the above-mentioned at least one morphological error may include at least one of a chamfer distance error, a model edge length error, and a model surface normal vector error.

[0225] In step S902, a second error function for characterizing the model error is constructed based on the at least one morphological error.

[0226] In this embodiment, after at least one morphological error between the reference facial model and the preset standard model is determined, a second error function for characterizing the model error can be constructed based on the at least one morphological error.

[0227] In some embodiments, after determining the at least one morphological error mentioned above, the second error function can be determined based on the weighted summation result of the at least one morphological error to characterize the model error between the reference facial model and the preset standard model based on the function.

[0228] For example, assuming that the at least one morphological error includes a chamfer distance error l chamfer , model side length error l edge And the model surface normal vector error l normal , the second error function C for solving the second three-dimensional feature point set Qs′ corresponding to the preset standard model can be constructed based on the following formula (9-1): detail Qs′ i ):

[0229] C detail (Qs′ i )=l chamfer +λ e l edge +λ n l normal ; (9-1)

[0230] In the above formula: e , n is the weight of the corresponding error, and its value can be an empirical value of 0.1.

[0231] In some embodiments, the chamfer distance error l chamfer It can be determined based on the following formula (9-2):

[0232]

[0233] From the above formula (9-2), we can know that the chamfer distance error l chamferThe value of consists of two parts (i.e., the two parts before and after the “+”): the first half represents each 3D feature point on the reference facial model To each 3D feature point Qs′ on the preset standard model j The sum of the minimum distances between them; the second half represents each three-dimensional feature point Qs′ on the preset standard model i To each 3D feature point on the reference face model The minimum distance between .

[0234] In some embodiments, the above model side length error l edge It can be determined based on the following formula (9-3):

[0235] l edge =∑Qs′ i ∈Qs′(‖∑k j ∈N(Qs′ i )(Qs′ i ―k j )|‖ 2 ); (9-3)

[0236] Among them, N(Qs′ i ) is the point Qs′ i Adjacent connection points. From the above formula (9-3), we can see that the model side length error l edge In fact, it represents the sum of the vector distances between each point on the preset standard model and its adjacent connection points. detail (Qs′ i ) process, the distance between the adjusted vertices of the preset standard model and its adjacent connection points can be guaranteed to remain unchanged, thereby preventing the occurrence of abnormal vertices, that is, preventing the generation of very long connection edges.

[0237] In some embodiments, the above model surface normal vector error l normal It can be determined based on the following formula (9-4):

[0238]

[0239] Among them, n q Qs u Middle distance Qs′ i The normal vector of the nearest point q, <·> represents the vector inner product operation. From the above formula (9-4), we can see that the model surface normal vector error l normal Can be used to determine Qs′ i Is the surrounding local area parallel to Qs? u The corresponding model surface.

[0240] On this basis, Fig. 9Bis a flowchart showing how to obtain the target facial model according to an exemplary embodiment of the present disclosure; Fig. 9B As shown, the step S802 described above of adjusting the second three-dimensional feature points on the preset standard model based on the model error to obtain the target facial model may include the following steps S903-S904:

[0241] In step S903, solving the second error function to obtain a function solution result;

[0242] In step S904, the second three-dimensional feature points are adjusted based on the function solution result to obtain the target facial model.

[0243] In this embodiment, after constructing a second error function for characterizing the model error based on the at least one morphological error, the second error function can be solved based on a preset function solving method to obtain a function solving result, and then the second three-dimensional feature point can be adjusted based on the obtained function solving result to obtain the target facial model.

[0244] Among them, the above-mentioned preset function solving method can be selected from relevant technologies based on actual business needs, and this embodiment does not limit this.

[0245] In some embodiments, after obtaining the solution result of the second error function, the second three-dimensional feature points on the preset standard model can be adjusted based on the solution result, so that the adjusted preset standard model can be used as a reference facial model. After solving the second error function shown in the above formula (9-1) to obtain the function solution result, each second three-dimensional feature point on the preset standard model can be adjusted according to the obtained function solution result, so that the adjusted preset standard model can be used as the target facial model.

[0246] Fig. 9C is a schematic diagram of a facial scanning model according to an exemplary embodiment of the present disclosure; Fig.9D is a schematic diagram of a preset standard model according to an exemplary embodiment of the present disclosure; Fig.9E is a schematic diagram of a reference facial model according to an exemplary embodiment of the present disclosure; Fig.9F is a schematic diagram of a target facial model according to an exemplary embodiment of the present disclosure. FIG. 9C to FIG. 9F As shown, the facial scanning model is obtained by three-dimensional reconstruction based on the original image set of the target face, and the reference facial model is obtained by coarse alignment of the facial scanning model with the preset standard model. Therefore, it can retain more details of the target face, so it can be used as a fine alignment standard for the preset standard model, that is, by finely aligning the preset standard model with the reference facial model, the target facial model can be obtained.

[0247] As can be seen from the above description, this embodiment determines at least one morphological error between the reference facial model and the preset standard model, and constructs a second error function for characterizing the model error based on the at least one morphological error, then solves the second error function to obtain a function solution result, and then adjusts the second three-dimensional feature point based on the function solution result to obtain the target facial model. This can realize automatic acquisition of facial models without relying on manual operation for model alignment, can reduce the technical level requirements for the production personnel, can ensure the accuracy or precision of the acquired target facial model, and can save the operator a lot of time and energy, thereby improving the efficiency of facial model acquisition.

[0248] Fig.10 is a block diagram of a facial model acquisition device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be implemented by software and / or hardware, and can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablet computers, etc.), wearable devices (such as glasses, watches, etc.), etc. Specifically, Fig.10 As shown, the device includes: a construction and extraction module 110, a feature point determination module 120, a model generation module 130 and a model acquisition module 140, wherein:

[0249] Constructing an extraction module 110, for constructing a face scanning model and extracting two-dimensional facial feature points based on the original image set in response to acquiring an original image set of multiple frames of original images containing a target face;

[0250] A feature point determination module 120, configured to determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point;

[0251] A model generation module 130, configured to generate a reference facial model based on the facial scanning model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model;

[0252] The model acquisition module 140 is used to align the preset standard model with the reference facial model to obtain a target facial model.

[0253] As can be seen from the above description, the device of this embodiment responds to obtaining an original image set of multiple frames of original images containing a target face, constructs a facial scanning model based on the original image set and extracts facial two-dimensional feature points, and determines the first three-dimensional feature points on the facial scanning model based on the facial two-dimensional feature points, and then generates a reference facial model based on the facial scanning model, the first three-dimensional feature points and the second three-dimensional feature points on a preset standard model, and then aligns the preset standard model with the reference facial model to obtain a target facial model. This can realize automated acquisition of facial models without relying on manual operation for model alignment, thereby reducing the technical level requirements for the production personnel, thereby ensuring the accuracy or precision of the acquired target facial model, and saving a lot of time and energy of the operator, thereby improving the processing efficiency of the facial model.

[0254] Fig.11 2 is a block diagram of another facial model acquisition device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be implemented by software and / or hardware, and can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablet computers, etc.), wearable devices (such as glasses, watches, etc.), etc. Among them, the construction extraction module 210, the feature point determination module 220, the model generation module 230 and the model acquisition module 240 are similar to the aforementioned Fig.10 The functions of the construction extraction module 110, the feature point determination module 120, the model generation module 130 and the model acquisition module 140 in the illustrated embodiment are the same and will not be described in detail herein.

[0255] like Fig.11 As shown, constructing the extraction module 210 may include:

[0256] The model building unit 211 is used to input the original image set into a preset three-dimensional reconstruction algorithm model to obtain the facial scanning model.

[0257] In some other embodiments, the construction and extraction module 210 may include: a feature extraction unit 212;

[0258] The feature extraction unit 212 may be used to:

[0259] Selecting at least two frames of target images from the original image set, wherein the orientation of the target face in the target images is a preset orientation;

[0260] Based on a preset deep learning model, feature points are extracted from the at least two frames of target images respectively to obtain the facial two-dimensional feature points.

[0261] In some embodiments, the feature point determination module 220 may include:

[0262] A second determining unit 221, configured to determine a third three-dimensional feature point corresponding to the two-dimensional facial feature point based on a triangulation method;

[0263] The feature point determination unit 222 is used to align the third three-dimensional feature points onto the facial scanning model to obtain the first three-dimensional feature points.

[0264] In some embodiments, the second determining unit 221 may also be used to:

[0265] Normalizing the pixel coordinates of the two-dimensional facial feature points on the target image of each frame based on a predetermined camera internal parameter to obtain the normalized coordinates of the two-dimensional facial feature points corresponding to the target image of each frame;

[0266] The third three-dimensional feature point is determined based on the normalized coordinates and a predetermined camera pose corresponding to each frame of the target image.

[0267] In some embodiments, the feature point determination unit 222 may also be used to:

[0268] Determining the Euclidean distance between the third three-dimensional feature point and each point on the facial scan model;

[0269] The first three-dimensional feature point is determined based on the point on the facial scan model whose Euclidean distance meets the set condition.

[0270] In some embodiments, the model generation module 230 may include:

[0271] An alignment unit 231, configured to align the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result;

[0272] The generating unit 232 is configured to adjust the first three-dimensional feature points on the facial scan model based on the alignment result to obtain the reference facial model.

[0273] In some embodiments, the alignment result includes a scale factor between the facial scanning model and the preset standard model and a pose transformation matrix from the facial scanning model to the preset standard model;

[0274] Furthermore, the alignment unit 231 may also be used for:

[0275] Constructing a first error function based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix;

[0276] The first error function is solved based on a preset optimization algorithm to obtain the scale factor and the posture transformation matrix.

[0277] In some embodiments, the model acquisition module 240 may include:

[0278] An error determination unit 241, configured to determine a model error between the preset standard model and the reference facial model;

[0279] The model acquisition unit 242 is used to adjust the second three-dimensional feature points on the preset standard model based on the model error to obtain the target facial model.

[0280] In some embodiments, the error determination unit 241 may also be used to:

[0281] Determine at least one morphological error between the reference facial model and the preset standard model, wherein the at least one morphological error comprises at least one of a chamfer distance error, a model side length error, and a model surface normal vector error;

[0282] constructing a second error function for characterizing the model error based on the at least one morphological error;

[0283] On this basis, the model acquisition unit 242 can also be used to:

[0284] Solving the second error function to obtain a function solution result;

[0285] The second three-dimensional feature points are adjusted based on the function solution result to obtain the target facial model.

[0286] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0287] Fig.12 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0288] Reference Fig.12 , device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0289] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0290] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0291] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 900.

[0292] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0293] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0294] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0295] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display panel and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0296] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0297] In an exemplary embodiment, the device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0298] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0299] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0300] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A facial model acquisition method, characterized in that: The method comprises: In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points; Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point; Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model; The preset standard model is aligned with the reference facial model to obtain a target facial model.

2. The method according to claim 1, characterized in that: The step of constructing a facial scanning model based on the original image set comprises: The original image set is input into a preset three-dimensional reconstruction algorithm model to obtain the facial scanning model.

3. The method according to claim 1, characterized in that The extracting of facial two-dimensional feature points based on the original image set includes: Selecting at least two frames of target images from the original image set, wherein the orientation of the target face in the target images is a preset orientation; Based on a preset deep learning model, feature points are extracted from the at least two frames of target images respectively to obtain the facial two-dimensional feature points.

4. The method according to claim 1, characterized in that: The determining of the first three-dimensional feature point on the facial scanning model based on the two-dimensional facial feature point comprises: Determine a third three-dimensional feature point corresponding to the two-dimensional facial feature point based on a triangulation measurement method; The third three-dimensional feature points are aligned to the facial scanning model to obtain the first three-dimensional feature points.

5. The method according to claim 4, characterized in that The determining of the third three-dimensional feature points corresponding to the two-dimensional facial feature points based on the triangulation method includes: Normalizing the pixel coordinates of the two-dimensional facial feature points on the target image of each frame based on a predetermined camera internal parameter to obtain the normalized coordinates of the two-dimensional facial feature points corresponding to the target image of each frame; The third three-dimensional feature point is determined based on the normalized coordinates and a predetermined camera pose corresponding to each frame of the target image.

6. The method according to claim 4, characterized in that The step of aligning the third three-dimensional feature points onto the facial scanning model to obtain the first three-dimensional feature points includes: Determining the Euclidean distance between the third three-dimensional feature point and each point on the facial scan model; The first three-dimensional feature point is determined based on the point on the facial scan model whose Euclidean distance meets the set condition.

7. The method according to claim 1, characterized in that The step of generating a reference facial model based on the facial scanning model, the first three-dimensional feature points, and second three-dimensional feature points on a preset standard model includes: Aligning the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result; The first three-dimensional feature points on the facial scan model are adjusted based on the alignment result to obtain the reference facial model.

8. The method according to claim 7, characterized in that The alignment result includes a scale factor between the facial scanning model and the preset standard model and a pose transformation matrix from the facial scanning model to the preset standard model; The aligning the first three-dimensional feature point with the second three-dimensional feature point to obtain an alignment result includes: Constructing a first error function based on the first three-dimensional feature points, the second three-dimensional feature points, the scale factor, and the pose transformation matrix; The first error function is solved based on a preset optimization algorithm to obtain the scale factor and the posture transformation matrix.

9. The method according to claim 1, characterized in that: The step of aligning the preset standard model with the reference facial model to obtain a target facial model includes: Determining a model error between the preset standard model and the reference facial model; The second three-dimensional feature points on the preset standard model are adjusted based on the model error to obtain the target facial model.

10. The method according to claim 9, characterized in that The determining of a model error between the preset standard model and the reference facial model comprises: Determine at least one morphological error between the reference facial model and the preset standard model, wherein the at least one morphological error comprises at least one of a chamfer distance error, a model side length error, and a model surface normal vector error; constructing a second error function for characterizing the model error based on the at least one morphological error; The step of adjusting the second three-dimensional feature points on the preset standard model based on the model error to obtain the target facial model includes: Solving the second error function to obtain a function solution result; The second three-dimensional feature points are adjusted based on the function solution result to obtain the target facial model.

11. A facial model acquisition device, characterized in that: The device comprises: Constructing an extraction module, which is used to construct a face scanning model and extract two-dimensional facial feature points based on the original image set in response to obtaining a set of original images of multiple frames containing the target face; A feature point determination module, configured to determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point; A model generation module, used to generate a reference facial model based on the facial scanning model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model; The model acquisition module is used to align the preset standard model with the reference facial model to obtain a target facial model.

12. An electronic device, characterized in that: The device comprises: a processor and a memory for storing a computer program; Wherein, the processor is configured to implement, when executing the computer program: In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points; Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point; Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model; The preset standard model is aligned with the reference facial model to obtain a target facial model.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it realizes: In response to acquiring an original image set of multiple frames of original images containing a target face, constructing a facial scanning model based on the original image set and extracting two-dimensional facial feature points; Determine a first three-dimensional feature point on the facial scan model based on the two-dimensional facial feature point; Generate a reference facial model based on the facial scan model, the first three-dimensional feature points and second three-dimensional feature points on a preset standard model; The preset standard model is aligned with the reference facial model to obtain a target facial model.