Identifying facial feature point locations for AI systems and applications

Abstract

The invention relates to recognition of facial feature point locations for AI systems and applications. In various examples, feature point identification and redirection for AI systems and applications are described herein. Systems and methods for determining locations of corresponding facial feature points on a second 3D face (e.g., a target facial mesh) using a first three-dimensional (3D) face (e.g., a deformed model mesh) that has been associated with the locations of the facial feature points are disclosed. To determine the position, one or more iterations of a transformation process and / or a fitting process may be performed on the first 3D face in order to deform feature points of the first 3D face to align with second feature points on the second 3D face. After performing an iteration of the transformation process and / or fitting process, a location (e.g., vertex) on the second 3D face closest to a feature point location (e.g., vertex) on the first 3D face is identified and used as a location of a corresponding facial feature point on the second 3D face.

Description

Background Art

[0001] Many applications, such as gaming applications, interactive applications, communication applications, multimedia applications, etc., use animated characters or digital avatars (humanoid or otherwise) that interact with users of the application and / or other animated characters within the application. In order to provide a more realistic experience to the user, some animated characters interact using both audio (e.g., voice) and visual indicators. For example, when the animated character interacts with the user, the application may synchronize the lip movements of the animated character with the voice being output by the animated character, while also causing the animated character to visually express emotions using facial / head movements. Visually expressing facial emotions may include causing the animated character to move various features of the face, such as the eyes, eyebrows, forehead, face, nose, cheeks, and / or other features of the face.

[0002] Therefore, many methods have been developed to determine the locations of various feature points on an animated face (e.g., lips, eyes, nose, ears, cheeks, hairline, jawline, etc.), which are then used for animation at these locations. For example, these methods use a model to first perform two-dimensional (2D) feature point detection on an animated character, and then use a camera to project the feature points onto the animated character. While these methods provide accurate results for feature point detection in some cases (e.g., when the animated character includes a symmetrical human face for which the model was trained), the accuracy of these methods is reduced in other cases, such as when there is a different camera angle relative to the animated character, the lighting changes, and / or the animated character does not contain much texture. In addition, these methods are inaccurate for certain types of animated characters (e.g., cartoon characters and / or animated faces for which the model was not trained). For example, when trying to transfer learned facial movements of a humanoid image to a cartoon image, feature points corresponding to the humanoid image may not align well with corresponding features of the cartoon image. Therefore, the transfer of facial landmarks may be completely inaccurate, resulting in a large amount of manpower required to rearrange and realign the facial landmarks on the cartoon character. Summary of the invention

[0003] Embodiments of the present disclosure relate to identifying facial and / or other feature point locations for AI systems and applications. Systems and methods are disclosed for determining the locations of corresponding facial feature points on a second 3D face (e.g., a target face mesh, also referred to as a "target face") using a first three-dimensional (3D) face (e.g., a morphable model mesh) that has been associated with the locations of the facial feature points. In order to determine the location, one or more iterations of one or more fitting processes (e.g., one or more 3D morphable model fitting processes) may be performed on the first 3D face to morph the first 3D face to be similar to the second 3D face. For example, the first fitting process may include a rigid transformation (e.g., translation, rotation, scaling, etc.) of the first 3D face to align the first 3D face with respect to the second 3D face. In addition, the second fitting process may include optimizing a weighted point distance between the first 3D face and the second 3D face. After performing one or more iterations of the fitting process, the locations (e.g., vertices) on the second 3D face that are closest to the locations (e.g., vertices) of the feature points on the first 3D face are identified and used as the locations of the corresponding facial feature points on the second 3D face.

[0004] Compared to conventional systems (e.g., systems that rely on 2D to 3D conversion of feature points), the systems and methods of the present disclosure are able to better perform feature point detection on a target face by deforming a first 3D face into a target face using the process described herein. For example, conventional systems use models trained on specific human faces to perform 2D feature point detection. As such, these models are less accurate when processing faces from different camera angles, with different lighting, and / or when animated characters do not contain textures. In addition, these models are less accurate when processing other types of faces (e.g., cartoon character faces) for which the models were not trained. In contrast, by initially deforming the first 3D face into a target face, in some embodiments, the current system is still able to perform feature point detection with high accuracy when the camera angle changes, the lighting changes, the animated character does not contain textures, and / or for new types of animated characters (e.g., cartoon characters). BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present system and method for identifying facial and / or other feature point locations for AI systems and applications is described in detail below with reference to the accompanying drawings, wherein:

[0006] Figure 1A An example data flow diagram showing a process of automatically identifying locations of feature points on a face according to some embodiments of the present disclosure;

[0007] Figure 1BAn example data flow diagram showing a process of interactively identifying locations of feature points on a face according to some embodiments of the present disclosure;

[0008] Figure 2A shows an example of a deformable face according to some embodiments of the present disclosure;

[0009] Figure 2B An example of a target face for determining feature point locations according to some embodiments of the present disclosure is shown;

[0010] Figure 3 shows an example of orienting a first face relative to a second face according to some embodiments of the present disclosure;

[0011] Figure 4 shows an example of aligning a first face with respect to a second face using an enclosing shape according to some embodiments of the present disclosure;

[0012] Figure 5 shows an example of indicating one or more marker locations associated with one or more feature points on a face according to some embodiments of the present disclosure;

[0013] Figure 6A-6B An example of performing multiple iterations of rigid transformation and structure fitting to make a first face better represent a second face is shown according to some embodiments of the present disclosure;

[0014] Figure 7 An example of using the locations of feature points on a first face to determine the locations of feature points on a second face according to some embodiments of the present disclosure is shown;

[0015] Figure 8 shows an example of indicating a marker location associated with a feature point on a face according to some embodiments of the present disclosure;

[0016] Fig. 9 shows an example of deforming a first face to represent a second face using a marker location according to some embodiments of the present disclosure;

[0017] Fig.10 A flow chart illustrating a method for identifying locations of feature points on a face according to some embodiments of the present disclosure is shown;

[0018] Fig.11 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0019] Fig.12 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Systems and methods for identifying feature point locations on avatars, digital characters, and / or other actors or objects for use in AI systems and applications are disclosed. Although primarily described with respect to facial feature point recognition and / or transfer (e.g., from one "face" to another "face"), this is not intended to be limiting, and feature points of other features (e.g., arms, legs, torsos, etc., as well as non-body related feature points) may be recognized and / or transferred without departing from the scope of the present disclosure.

[0021] With respect to implementations of faces, for example, one or more systems may receive, obtain, generate, and / or retrieve first data representing a first three-dimensional (3D) face (also referred to simply as a "first face") associated with feature point locations. As described herein, the first face may correspond to a morphable model face, where the locations of the feature points are known, such as by using user input, machine learning, and / or any other technique for determining the locations of feature points on a face. In addition, the feature points may be associated with features of a face, such as, but not limited to, one or more eye features (e.g., top of eye, bottom of eye, side of eye, middle of eye), one or more lip features (e.g., top of lip, bottom of lip, side of lip, middle of lip, etc.), one or more cheek features, one or more nose features, one or more ear features, one or more chin features, one or more hairline features, one or more forehead features, and / or any other features of a face (or other features or body parts).

[0022] One or more systems may also receive, obtain, generate and / or retrieve second data representing a second 3D face (also referred to simply as a "second face" or "target face") for which corresponding locations of feature points are to be determined. In some examples, the second face may include a structure similar to the first face. For example, the second face and the first face may both include a human face with similar features, such as a mouth, eyes, nose, chin, cheeks, ears, etc. However, in other examples, the second face may include a different structure than the first face. For example, the second face may include a cartoon character face, while the first face includes a human face. In such examples, the second face may not include one or more features of the first face, such as a nose or an eye. In any of these examples, the system may then process the second face with respect to the first face to determine the locations of the feature points.

[0023] For example, one or more systems may initially align a first face with respect to a second face. In some examples, the system aligns the first face with respect to the second face by matching the orientation of the first face with the orientation of the second face. In such examples, the system may use directions associated with the face to match these orientations. For example, the user may indicate and / or the system may automatically determine a direction outward from the face (e.g., indicating a forward direction of the face) and / or a direction outward from the top of the head associated with the face, and then use these directions to match these orientations. The system may also initially scale the first face with respect to the second face. For example, the system may determine a first enclosing shape associated with the first face (e.g., a first enclosing box) and a second enclosing shape associated with the second face (e.g., a second enclosing box). As described herein, an enclosing shape may include dimensions of at least three directions, such as an x-direction, a y-direction, and a z-direction. The system may then scale the first face with respect to the second face by scaling the first enclosing shape to match the second enclosing shape.

[0024] In some examples, one or more systems may use one or more additional and / or alternative processes to further align the first face with the second face. For example, the system may receive one or more inputs indicating one or more locations of one or more feature points associated with the second face (e.g., one or more guide points). For example, the system may receive input indicating the location of a nose (and / or nose features), the location of an eye (and / or eye features), the location of a mouth (and / or mouth features), etc. on the second face that matches the corresponding location of a corresponding feature point of the first face. In some examples, the system may further deform the first face based at least on the one or more locations of the indicated one or more feature points. For example, the system may move one or more locations of one or more feature points associated with the first face to match one or more locations of the indicated one or more feature points on the second face.

[0025] One or more systems may then perform one or more iterations of one or more processes to deform the first face to be similar to the second face. For example, the system may perform a first process that includes a rigid transformation in which the first face is translated, rotated, and / or scaled to better represent the second face. In some examples, the system uses one or more functions to perform the rigid transformation. The system may then perform a second process that includes model fitting in which the structure of the first face is updated to better represent the second face. For example, model fitting may include updating one or more vertex positions associated with the first face to match one or more vertex positions associated with the second face. Additionally, in some examples, the system may perform model fitting using one or more functions. In some embodiments, but not limited to, model fitting may include one or more 3D deformation model fitting processes.

[0026] Then, one or more systems may continue to repeatedly perform the rigid transformation and / or model fitting using multiple iterations. In some examples, the system may use one or more techniques to determine the number of iterations. For the first example, the system may use a set of iterations, such as but not limited to one iteration, two iterations, five iterations, ten iterations, and / or any other number of iterations. For the second example, the system may determine the number of iterations based on the result associated with the function. For example, the system may continue to perform the rigid transformation and / or model fitting until a first value associated with the first function for the rigid transformation and / or a second value associated with the second function for model fitting meets (e.g., is less than or equal to) one or more thresholds. In any example, after performing these processes, the first face may substantially represent the second face. For example, the translation, rotation, scaling, and / or shape of the first face may substantially match the translation, rotation, scaling, and / or shape of the second face.

[0027] Then, one or more systems can use the locations of the feature points on the first face to determine the locations of corresponding feature points on the second face. For example, the system can use the locations of eye features, lip features, cheek features, nose features, ear features, chin features, etc. on the first face to determine the locations of eye features, lip features, cheek features, nose features, ear features, chin features, etc. on the second face. In some examples, the system can perform one or more techniques to determine the locations of corresponding feature points on the second face.

[0028] For example, for a feature point, the system may determine a point on the second face that is near the feature point on the first face. In some examples, the system may determine that the point is near the feature point based at least on the point including the closest point on the second face compared to the feature point. Additionally, or alternatively, in some examples, the system may determine that the point is near the feature point based at least on the point being within a threshold distance to the feature point. The system may then determine a first normal direction associated with the point and a second normal direction associated with the feature point. In addition, when the angle between the first normal direction and the second normal direction is within a threshold angle (e.g., 10 degrees, 30 degrees, 45 degrees, 70 degrees, etc.), the system may determine that the point corresponds to the feature point, or when the angle between the first normal direction and the second normal direction exceeds the threshold angle, the system may determine that the point does not correspond to the feature point.

[0029] If the system determines that the point corresponds to a feature point, the system may determine that the feature point located at the feature point on the first face is also located at the point on the second face. However, if the system determines that the point does not correspond to a feature point, the system may perform one or more similar processes using one or more additional points on the second face (e.g., using the next closest point). The system may then continue to perform similar processes using the locations of the feature points on the first face to continue to determine corresponding locations of the feature points on the second face. In addition, the system may output data representing the locations of the feature points on the second face.

[0030] In some examples, the system may further use one or more techniques to verify the location of the feature points on the second face. For example, the user device may receive output data and use the output data to display content indicating the location of the feature points. For example, the content may include a second face showing the location of the feature points. Then, if one or more of the locations are inaccurate, the user may use the user device to update one or more of the locations. For example, if a feature point associated with one side of the mouth is indicated as being in the middle of the mouth, the user may move the feature point location to one side of the mouth. The system may then receive data from the user device, wherein the data represents any changes made to the location of the feature points on the second face, and update the location of the feature points on the second face based at least on the data.

[0031] The systems and methods described herein may be used for a variety of purposes, such as, but not limited to, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0032] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems implementing one or more large language models (LLMs), systems for performing conversational AI operations, systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0033] refer to Figure 1A , Figure 1A An example data flow diagram of a process for identifying feature point locations (in this example, specifically faces) according to some embodiments of the present disclosure is shown. It should be understood that this arrangement and other arrangements described herein are set forth as examples only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may also be used in addition to or in place of the arrangements and elements shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and position. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in a memory.

[0034] Process 100 may include an orientation component 102 receiving first facial data 104 representing a first three-dimensional (3D) face (also referred to simply as a "first face" or "morphable face") associated with feature point locations. As described herein, the first face may correspond to a morphable model face for which the locations of the feature points are known, such as by using user input, machine learning, and / or any other technique for determining feature point locations on a face. Additionally, the feature points may be associated with features of the first face, such as, but not limited to, one or more eye features (e.g., top of eye, bottom of eye, side of eye, middle of eye), one or more lip features (e.g., top of lip, bottom of lip, side of lip, middle of lip, etc.), one or more cheek features, one or more nose features, one or more ear features, one or more chin features, and / or any other features of the face.

[0035] In some examples, the first facial data 104 may represent the first face as a set of points (e.g., vertices) located on the first face. For example, the first facial data 104 may represent coordinates associated with the points, such as an x-coordinate position, a y-coordinate position, and / or a z-coordinate position associated with the points. Additionally, the positions of feature points may be associated with points and / or coordinates. For the first example, the positions of feature points may be associated with points on a first face that includes features corresponding to the feature points. For the second example, the positions of feature points may be associated with coordinates, such as an x-coordinate position, a y-coordinate position, and / or a z-coordinate position.

[0036] For example, Figure 2A An example of a deformable face 202 (e.g., the first face described herein) is shown according to some embodiments of the present disclosure. As shown, the first face 202 can include multiple locations of multiple feature points 204 (although only one is labeled for clarity). For example, the feature points 204 can be associated with the mouth (e.g., the bottom, sides, and top of the mouth), the cheeks, the chin, the nose (e.g., the middle and sides of the nose), the first eye (e.g., the bottom, sides, and top of the left eye), the second eye (e.g., the bottom, sides, and top of the right eye), the center between the two eyes, the eyebrows, and the ears. Although Figure 2A The example of shows several locations of several feature points 204 associated with the first face 202, but in other examples, the first face 202 can be associated with additional and / or alternative feature points.

[0037] Return to reference Figure 1A In an example of, the process 100 may also include the orientation component 102 receiving second facial data 106, which represents a second 3D face (also referred to as just "second face" or "target face") whose corresponding positions of feature points are to be determined. In some examples, the second face may include a structure similar to the first face. For example, both the second face and the first face may include a human face with similar features, such as a mouth, eyes, nose, chin, cheeks, ears, etc. However, in other examples, the second face may include a different structure than the first face. For example, the second face may include a cartoon character face, while the first face again includes a human face. In such an example, the second face may not include one or more features of the first face, such as a nose or an eye. In some examples, similar to the first facial data 104, the second facial data 106 may represent the second face as a set of points (e.g., vertices) located on the second face. For example, the second facial data 106 may represent coordinates associated with these points, such as x-coordinate positions, y-coordinate positions, and / or z-coordinate positions associated with these points.

[0038] For example, Figure 2BAn example of a target face 206 (e.g., a second face described herein) for determining feature point locations according to some embodiments of the present disclosure is shown. As shown, the second face 206 can be similar to the first face 202 in that both the second face 206 and the first face 202 include human faces. For example, the second face 206 can include features similar to the first face 202, such as, but not limited to, lips, nose, cheeks, chin, eyes, ears, eyebrows, etc. However, in other examples, the second, target face can include a different type of face than the first face 202. For example, the second face can include a cartoon face (e.g., an animal face, an alien face, a cartoon face, a robot face, etc.) that does not include one or more features of the first face 202.

[0039] Return to reference Figure 1A In an example of, process 100 may include orientation component 102 orienting a first face with respect to a second face. As described herein, in some examples, orientation component 102 may orient the face using one or more directions associated with the face. For example, orientation component 102 may orient the face based at least on matching one or more first directions associated with the first face with one or more second directions associated with the second face. As described herein, the direction may include, but is not limited to, the direction of the face, the direction of the top of the head associated with the face, the direction of the ear, and / or any other direction. In some examples, orientation component 102 may determine the direction using input data 108 received from one or more user devices 110. For example, input data 108 may represent a direction indicated by one or more users. Additionally or alternatively, in some examples, orientation component 102 may determine the direction based on processing the face, such as by using one or more models trained to determine the direction.

[0040] For example, Figure 3 206 according to some embodiments of the present disclosure. As shown, the orientation component 102 can determine at least a lateral direction 302 (e.g., along the x-axis) and a vertical direction 304 (e.g., along the y-axis) associated with the first face 202, as well as a forward or inward direction (e.g., along the z-axis, due to the Figure 3In some examples, the first face 202 is directed to the second face 206, and the first face 202 is directed to the second face 206. In some examples, the first face 202 is directed to the second face 206, and the second face 206 is directed to the first face 202. In some examples, the first face 202 is directed to the second face 206, and the second face 206 is directed to the second face 206 ...

[0041] Return to reference Figure 1A In an example of, process 100 may include an enclosing component 112 determining a first enclosing shape (e.g., a first enclosing box) associated with a first face and a second enclosing shape (e.g., a second enclosing box) associated with a second face. As described herein, in some examples, the enclosing shapes may include dimensions in at least three directions, such as an x-direction, a y-direction, and a z-direction. Additionally, in some examples, the enclosing shapes may be generated so that they include the same parts of the character associated with the faces (e.g., a head, a neck, and / or other parts). The enclosing component 112 may then scale the first face with respect to the second face using the enclosing shapes. For example, the enclosing component 112 may scale the first face with respect to the second face by aligning the first enclosing shape with respect to the second enclosing shape.

[0042] For example, Figure 4 An example of aligning the first face 202 with respect to the second face 206 using an enclosing shape according to some embodiments of the present disclosure is shown. Figure 4 As shown in the example of FIG. 1 , the enclosure component 112 can determine a first enclosure shape 402 associated with the first face 202 and a second enclosure shape 404 associated with the second face 206. Figure 4 The example of FIG. 4 shows that the enclosing shapes 402 and 404 are two-dimensional, for example, by having dimensions in the x-direction and the y-direction, but based on the enclosing shapes also including a dimension in the z-direction, the enclosing shapes 402 and 404 can be three-dimensional. In some examples, the enclosing component 112 can use one or more models to determine one or more of the enclosing shapes 402 and 404. In some examples, the enclosing component 112 can use one or more inputs from a user to determine one or more of the enclosing shapes 402 and 404.

[0043] Then, the enclosing component 112 may align the first face 202 with respect to the second face 206 using the enclosing shapes 402 and 404. For example, the enclosing component 112 may align the first face 202 with respect to the second face 206 by changing the scale of the first enclosing shape 402 to substantially match the scale of the second enclosing shape 404. Thus, since the portion of the first character associated with the first face 202 included in the first enclosing shape 402 and the portion of the second character associated with the second face 206 included in the second enclosing shape 404 are the same portion, the first face 202 should be scaled to the second face 206 after the alignment is performed.

[0044] Return to reference Figure 1A In an example of a second face, process 100 may include a user device 110 providing input data 108, the input data 108 representing one or more "marker" locations of one or more feature points associated with the second face. For example, a user interface may be provided to the user, the user interface depicting at least a first face, one or more locations of one or more feature points on the first face, and a second face. The user may then use the user interface to indicate one or more corresponding locations of the one or more feature points on the second face. For example, if the user interface indicates the location of a feature point associated with the nose of the first face, the user may provide input indicating the location of the nose on the second face. In some examples, the number of marker locations may be determined based at least on one or more factors. For example, the number of marker locations may be determined based at least on user input (e.g., which indicates the number of marker locations), the type of the second face, and / or any other factor. For example, if the two faces include the same type of face, such as a human face, a first number of marker locations may be used. However, if the two faces include different types of faces, such as a human face and a cartoon face, a second, greater number of marker locations may be used. In some embodiments, in addition to or in lieu of user input to the UI, feature point updates may be automatically performed using a trained model or algorithm that analyzes the projected feature points and shifts them to identified feature points (e.g., the left edge of the eye, the right edge of the eye, the tip of the nose, etc.).

[0045] For example, Figure 5 2 shows an example of indicating one or more marker locations associated with one or more feature points on the second face 206 according to some embodiments of the present disclosure. Figure 5 As shown in the example of , a first face 202 indicating the location of at least one feature point 204 may be provided to the user. Figure 5The example of shows the location of a single feature point 204, but in other examples, any number of feature point locations can be provided to the user. For example, the user can be provided with one feature point location, two feature point locations, five feature point locations, ten feature point locations, etc. The user can then indicate the location of the feature point 502 on the second face 206 that corresponds to the feature point 204. As described in more detail herein, other components (e.g., the transformation component 114, the fitting component 116, etc.) can then use this initial location of the feature point 502 to deform the first face 202 to be similar to the second face 206.

[0046] For example, and return a reference Figure 1A For example, process 100 may include transformation component 114 performing a rigid transformation, wherein the first face is translated, rotated, and / or scaled to better represent the second face. As described herein, translation may include about a first coordinate direction (e.g., Figure 6A-6B The first face is rotated to match the second face in the first coordinate direction. In addition, the rotation may include rotating the first face with respect to the second coordinate direction (e.g., Figure 6A-6B The first face may be rotated in the z-coordinate direction in the example of the embodiment of the present invention so that the first face is rotated to match the second face in the second coordinate direction. In addition, scaling may include changing the size of the first face so that the first face better matches the size of the second face.

[0047] In some examples, the transformation component 114 may use one or more functions to perform a rigid transformation. For example, the transformation component 114 may use the following functions to perform a rigid transformation:

[0048]

[0049] As shown in the figure, function (1) performs a rigid transformation by calculating the optimal rotation, optimal translation, and optimal scaling that minimizes the energy function E. For example, ppd() is the point p on the second face i and point q on the first face i The distance between the point and the plane, n i From point q i In addition, i may include the corresponding point index, such as the nearest point, which can be given by a kd-tree search of the nearest point.

[0050] For example, for point q on the first face i , the transformation component 114 may use one or more techniques to find the closest point p on the second face i Then, the transformation component 114 can determine the i The associated first normal vector and the i To point qi The second vector of (1). In addition, the transformation component 114 can determine the dot product between the vectors. In some examples, if the vectors include similar directions, the dot product of the two vectors can be close to a first value, such as 1. In addition, if the vectors include orthogonal directions, the dot product of the two vectors can be close to a second value, such as 0. Therefore, when the vectors are similar to each other, the effect of the vertex pair on the function (1) is greater than when the vectors are orthogonal. The transformation component 114 can then use these points to minimize the energy function E.

[0051] Process 100 may also include fitting component 116 updating the structure of the first face to better represent the second face. For example, fitting component 116 may update one or more locations of one or more points (e.g., one or more vertices) of the first face to match one or more points (e.g., one or more vertices) of the second face. In some examples, fitting component 116 may use one or more functions, such as the following functions, when updating the structure of the first face:

[0052]

[0053] In function (2), p i Still a point on the second face, (m+Ua) i is the facial reconstruction performed on the first face by the fitting component 116. Additionally, i may include corresponding point indices, such as the nearest point, which may be given by a kd-tree search of the nearest point. Furthermore, W is a per-correspondence weighting matrix that weights point distance errors based at least on the vertex normal direction. In other words, function (2) minimizes the distance between a point on the first face and a point on the second face.

[0054] For example, for point q on the first face i , the fitting component 116 may use one or more techniques to find the closest point p on the second face i The fitting component 116 can then determine the i The associated first normal vector and the i To point q i The second vector of . Additionally, the fitting component 116 can determine the dot product between the vectors. In some examples, if the vectors include similar directions, the dot product of the two vectors can be close to a first value, such as 1. Moreover, if the vectors include orthogonal directions, the dot product of the two vectors can be close to a second value, such as 0. Therefore, when the vectors are similar to each other, the effect of the vertex pair on function (2) is greater than when the vectors are orthogonal. The fitting component 116 can then use the included points to minimize the distance between the point on the first face and the point on the second face.

[0055] like Figure 1AAs further shown in the example of, process 100 can continue to repeat multiple iterations between the transformation component 114 performing the rigid transformation and the fitting component 116 performing the update of the first face. In some examples, process 100 can use a set number of iterations, such as but not limited to one iteration, two iterations, five iterations, ten iterations, and / or any other number of iterations. In some examples, the number of iterations can be based on the results associated with function (1) and / or function (2). For example, the transformation component 114 can continue to perform the rigid transformation and / or the fitting component 116 can continue to update the first face until the error E of function (1) meets (e.g., is less than or equal to) a threshold error (which can be represented by threshold data 118) and / or until function (2) meets (e.g., is less than or equal to) a threshold (which can also be represented by threshold data 118). In any example, after performing these processes, the feature point locations on the first face can substantially correspond to the feature point locations on the second face. For example, the translation, rotation, scale, and / or shape of the first face may be taken into account so that the feature points substantially match on the second face due to the translation, rotation, scale, and / or shape of the second face.

[0056] For example, Figure 6A-6B An example of performing multiple iterations of rigid transformation and structure fitting to make the first face 202 better represent the second face 206 according to some embodiments of the present disclosure is shown. Fig. 6A As shown in the example of , during the first iteration, the transformation component 114 can rotate 602 the first face 202 about a first coordinate direction (e.g., an x-coordinate direction) so that the first face 202 is rotated to match the second face 206 in the first coordinate direction. In addition, the transformation component 114 can rotate 604 the first face 202 about a second coordinate direction (e.g., a z-coordinate direction) so that the first face 202 is rotated to match the second face 206 in the second coordinate direction. In addition, the transformation component 114 can update the scale of the first face 202 so that the first face 202 better matches the size of the second face 206.

[0057] The fitting component 116 can then update the structure of the first face 202 to better represent the second face 206. For example, the fitting component 116 can update one or more positions of one or more points (e.g., one or more vertices) of the first face 202 to match one or more points (e.g., one or more vertices) of the second face 206. Fig. 6AAs shown in the example of , after the transformation component 114 performs a rigid transformation on the first face 202 and the fitting component 116 updates the structure of the first face 202, the first face 202 can begin to match the second face 206. For example, smooth areas without contours can indicate that the first face 202 matches the second face 206, while areas including contours may require more updates to make the first face 202 better match the second face 206.

[0058] For example, Figure 6B As shown in the example of , during the second iteration, the transformation component 114 can rotate 606 the first face 202 again about the first coordinate direction (e.g., the x-coordinate direction) so that the first face 202 is rotated to better match the second face 206 in the first coordinate direction. In addition, the transformation component 114 can rotate 608 the first face 202 again about the second coordinate direction (e.g., the z-coordinate direction) so that the first face 202 is rotated to better match the second face 206 in the second coordinate direction. In addition, the transformation component 114 can update the scale of the first face 202 again so that the first face 202 better matches the size of the second face 206.

[0059] The fitting component 116 can then update the structure of the first face 202 to better represent the second face 206. For example, the fitting component 116 can update one or more positions of one or more points (e.g., one or more vertices) of the first face 202 to match one or more points (e.g., one or more vertices) of the second face 206. Figure 6B As shown in the example of , after the transformation component 114 performs a rigid transformation on the first face 202 and the fitting component 116 updates the structure of the first face 202, the first face 202 can begin to better match the second face 206. For example, again, smooth areas without contours can indicate that the first face 202 matches the second face 206, while areas that include contours may require more updates to make the first face 202 better match the second face 206. In this way, since there are fewer areas that include contours, Figure 6B The first face 202 in the example may better represent the second face 206 .

[0060] As described herein, the transformation component 114 may continue to perform the rigid transformation and / or the fitting component 116 may continue to perform the update of the first face 202 for a plurality of iterations. In some examples, the transformation component 114 and / or the fitting component 116 may use a set number of iterations, such as, but not limited to, one iteration, two iterations, five iterations, ten iterations, and / or any other number of iterations. In some examples, the number of iterations may be based on the results associated with the function (1) and / or the function (2). For example, the transformation component 114 may continue to perform the rigid transformation and / or the fitting component 116 may continue to update the first face 202 until the error E of the function (1) satisfies (e.g., is less than or equal to) a threshold error and / or until the function (2) satisfies (e.g., is less than or equal to) a threshold. In any example, after performing these processes, the first face 202 may substantially represent the second face 206. For example, the translation, rotation, scale, and / or shape of the first face 202 may substantially match the translation, rotation, scale, and / or shape of the second face 206.

[0061] Return to reference Figure 1A In an example of, process 100 may include feature point component 120 using the location of feature points on the first face to determine the location of feature points on the second face. As described herein, the feature points may be associated with features of the second face, such as, but not limited to, one or more eye features (e.g., top of the eye, bottom of the eye, side of the eye, middle of the eye), one or more lip features (e.g., top of the lip, bottom of the lip, side of the lip, middle of the lip, etc.), one or more cheek features, one or more nose features, one or more ear features, one or more chin features, and / or any other features of the face. In some examples, feature point component 120 may use one or more techniques to determine the location of feature points on the second face.

[0062] For example, for a feature point, the feature point component 120 may identify the closest point on the second face that matches the point associated with the feature point on the first face. The feature point component 120 may then determine a first normal direction of a first vertex associated with the point on the first face and a second normal direction of a second vertex associated with the closest point on the second face. In addition, the feature point component 120 may then determine whether the second normal direction is similar to the first normal direction. In some examples, the feature point component 120 may determine that the second normal direction is similar to the first normal direction based at least on the angle between the second normal direction and the first normal direction satisfying (e.g., being less than or equal to) a threshold angle (e.g., 10 degrees, 30 degrees, 45 degrees, 70 degrees, etc.) (which is represented by the threshold data 118), or determine that the second normal direction is not similar to the first normal direction based at least on the angle between the second normal direction and the first normal direction not satisfying (e.g., being greater than) the threshold angle.

[0063] If the feature point component 120 determines that the second normal direction is similar to the first normal direction, the feature point component 120 can determine that the location on the second face associated with the closest point includes the feature point. However, if the feature point component 120 determines that the second normal direction is not similar to the first normal direction, the feature point component 120 can determine the next closest point on the second face to the point on the first face. Additionally, the feature point component 120 can perform a similar process to determine whether the third normal direction associated with the second closest point is similar to the first normal direction. The feature point component 120 can then continue to perform these processes until a feature point location on the second face that matches the feature point location on the first face is found. Additionally, the feature point component 120 can perform these processes for one or more (e.g., each) additional feature points.

[0064] For example, Figure 7 An example of determining the locations of feature points on the second face 206 using the locations of feature points on the first face 202 according to some embodiments of the present disclosure is shown. As shown, the feature point component 120 may perform one or more processes described herein to determine the locations of feature points 702 (although only a few are labeled for clarity) on the second face 206 using the locations of corresponding feature points 204 (although only a few are labeled for clarity). For example, for feature point 204(1), the feature point component 120 may initially identify points on the second face 206 that include the points that are closest to the points on the first face 202 associated with the locations of feature point 204(1). The feature point component 120 may then determine a first normal direction 704 associated with the points on the first face 202 and a second normal direction 706 associated with the points on the second face 206. In addition, the feature point component 120 may determine whether the second normal direction 706 is similar to the first normal direction 704.

[0065] As described herein, if the feature point component 120 determines that the second normal direction 706 is not similar to the first normal direction 704 (e.g., the angle between the second normal direction 706 and the first normal direction 704 is greater than a threshold angle), the feature point component 120 can identify the next closest point on the second face 206. Figure 7 As shown in the example of , if the feature point component 120 determines that the second normal direction 706 is similar to the first normal direction 704 (e.g., the angle between the second normal direction 706 and the first normal direction 704 is less than or equal to the threshold angle), the feature point component 120 can determine that the location of the point on the second face 206 includes the feature point 702 (1) corresponding to the feature point 204 (1). The feature point component 120 can then perform a similar process for one or more (e.g., each) other feature points 204.

[0066] Return to reference Figure 1A In some examples, process 100 may include outputting updated facial data 122 representing the locations of feature points on the second face. In some examples, one or more user devices 110 may then display the second face along with the locations of the feature points to one or more users using updated facial data 122. For example, user device 110 may display second face 206 to the user as well as Figure 7 The positions of the feature points 702 in the example of . The user may then be able to determine whether the positions of the feature points are correct and / or update one or more positions if one or more positions are incorrect.

[0067] Reference now Figure 1B For example, Figure 1B An example data flow diagram of a process 124 for interactively identifying feature point locations on a face according to some embodiments of the present disclosure is shown. As shown in the figure, Figure 1B The process 124 can be similar to Figure 1A Process 100 of the present invention, except that process 124 may include user device 110 providing marked input data 126 representing one or more "marked" locations of one or more feature points associated with the second face, and then deformation component 128 performs an initial deformation on the first face based on at least the one or more marked locations of the one or more feature points. As such, in some examples, process 124 may better determine the locations of feature points on certain types of faces, such as faces that include one or more additional and / or alternative features and / or different and / or alternative feature proportions compared to the first face represented by facial data 104 (e.g., cartoon faces).

[0068] For example, Figure 8 An example of indicating the location of a marker associated with a feature point on a face according to some embodiments of the present disclosure is shown. Figure 8 As shown in the example of , the second face 802 now includes a cartoon without one or more features (e.g., a nose) of the first face 202. As such, the user may wish to provide additional marker locations associated with the second face 802 so that the rest of the process 124 can better match the first face 202 to the second face 802. Figure 8 In the example of FIG. 1 , a first face 202 indicating the locations of at least five feature points 204 ( 1 )-( 5 ) may be provided to the user. Figure 8 The example of FIG. 2 shows the positions of five feature points 204 (1)-(5), but in other examples, any number of feature point positions may be provided to the user. For example, one feature point position, two feature point positions, five feature point positions, ten feature point positions, etc. may be provided to the user.

[0069] The user may then identify locations of feature points 804(1)-(5) on the second face 802 that correspond to the locations of feature points 204(1)-(5) on the first face 202. For example, the user may indicate that the location of feature point 804(1) corresponds to the location of feature point 204(1), the location of feature point 804(2) corresponds to the location of feature point 204(2), the location of feature point 804(3) corresponds to the location of feature point 204(3), the location of feature point 804(4) corresponds to the location of feature point 204(4), and the location of feature point 804(5) corresponds to the location of feature point 204(5). In some examples, the user may indicate the locations of feature points 804(1)-(5) using one or more techniques, such as selecting a location on the second face 802, moving a marker to a correct location on the second face 802, and / or using any other technique.

[0070] Return to reference Figure 1B In some examples, the deformation component 128 may deform the first face by at least updating the vertices associated with the locations of the feature points on the first face to more closely match the marker locations on the second face. In some examples, the deformation component 128 may use one or more functions to perform the deformation of the first face. For example, the deformation component 128 may use the following function to deform the first face:

[0071] p i =R(q i ) (3)

[0072] In function (3), the deformation function R can use the point p on the first face i and point q on the second face i The deformation component 128 may then use the trained deformation function R to deform one or more points (e.g., all points) on the first face. In some examples, the deformation component 128 may also use a function such as a thin plate spline function (TPS) (and / or any other function) as a kernel function that minimizes the bending energy of the deformation. For example, the deformation component 128 may use the following function:

[0073] U(x,p i )=||xp i || (4)

[0074] For example, Fig. 9An example of using a marker position to deform a first face 202 to represent a second face 802 according to some embodiments of the present disclosure is shown. As shown, the deformation component 128 can deform the first face 202 by moving one or more vertices based at least on the positions of feature points 804 (1)-(5) indicated by the user, where the deformation is indicated by at least arrows 902 (1)-(3). Thus, by performing this deformation, the first face 202 appears more similar to the second face 802.

[0075] Return to reference Figure 1B In the example of, after performing deformation, process 124 can again be similar to process 100, with the transformation component 114 performing a rigid transformation, the fitting component 116 updating the structure of the first face to better represent the second face, the feature point component 120 determining the locations of the feature points on the second face, and outputting updated facial data 122 representing the locations of the feature points on the second face.

[0076] Reference now Fig.10 , each block of the method 1000 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 1000 can also be embodied as computer usable instructions stored on a computer storage medium. The method 1000 can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, by way of example, with respect to Figure 1A and Figure 1B Method 1000 is described. However, the method 1000 may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0077] Fig.10A flow chart showing a method 1000 for identifying feature point locations on a face is shown according to some embodiments of the present disclosure. At block B1002, the method 1000 may include obtaining first data representing a first three-dimensional (3D) face associated with one or more first feature point locations. For example, the system may receive facial data 104 representing a first face. As described herein, the first face may correspond to a morphable model face, where the first feature point locations are known, such as by using user input, machine learning, and / or any other technique for determining feature point locations on a face. In addition, the feature points may be associated with one or more features of the first face, such as, but not limited to, one or more eye features (e.g., top of the eye, bottom of the eye, side of the eye, middle of the eye), one or more lip features (e.g., top of the lip, bottom of the lip, side of the lip, middle of the lip, etc.), one or more cheek features, one or more nose features, one or more ear features, one or more chin features, and / or any other features.

[0078] At block B1004, method 1000 may include obtaining second data representing a second 3D face. For example, the system may also receive facial data 106 representing the second face. In some examples, the second face may include similar structures as the first face. For example, the second face and the first face may both include similar features, such as a mouth, eyes, nose, chin, cheeks, ears, etc. However, in other examples, the second face may include different structures than the first face. For example, the second face may include additional, fewer, and / or alternative features compared to the first face and / or may include different feature proportions compared to the first face.

[0079] At block B1006, method 1000 may include determining a correspondence between the first 3D face and the second 3D face based at least on performing at least one of one or more transformation processes or one or more fitting processes associated with the first 3D face. For example, the system may process the first face using at least transformation component 114 and / or fitting component 116 (e.g., by using one or more iterations). In some examples, the system may further process the first face using orientation component 102, encirclement component 112, and / or deformation component 128. Based at least on the processing, the system may determine a correspondence between the first face and the second face. As described herein, the correspondence may be associated with deforming the first face to be similar to the second face in terms of size, pose, proportion, etc.

[0080] At box B1008, method 1000 may include: determining one or more second feature point locations associated with the second 3D face based at least on the correspondence and using the one or more first feature point locations. For example, the system may process the first face and the second face using the feature point component 120 to determine the second feature point location based at least on the correspondence and the first feature point location. As described herein, the feature point component 120 may determine the second feature point location based at least on one or more nearest points associated with the first feature point location and / or one or more surface normal directions associated with the nearest point. In some examples, the system may further verify the result of the second feature point location, such as by using feedback from one or more users.

[0081] In any embodiments, once feature points have been transferred from a first face to a second face, various operations may be performed using the transferred feature points. For example, facial retargeting may be used to transfer animations learned for a first face to a second face without requiring additional programming or learning specifically for the second face (although additional learning / training may be performed for the second face with the retargeting as a start). In other examples, the transfer of feature points may help animate a variety of different types of avatars, characters, non-player characters, etc. that may differ greatly in appearance without having to learn each different facial structure, layout, feature type, etc. separately. In this way, facial feature points from one face or class of faces may be quickly transferred to other faces to speed up the facial animation process for the transferred or target face. These animations may be used for avatars, characters, etc. in any use case, such as gaming applications, video conferencing applications, metaverse / omniverse applications, etc.

[0082] Example computing device

[0083] Fig.111 is a block diagram of an example computing device 1100 suitable for implementing some embodiments of the present disclosure. The computing device 1100 may include an interconnect system 1102 that directly or indirectly couples the following devices: a memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communication interface 1110, an input / output (I / O) port 1112, an input / output component 1114, a power supply 1116, one or more presentation components 1118 (e.g., (one or more) displays), and one or more logic units 1120. In at least one embodiment, the (one or more) computing device 1100 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 1108 may include one or more vGPUs, one or more of the CPUs 1106 may include one or more vCPUs, and / or one or more of the logic units 1120 may include one or more virtual logic units. As such, computing device(s) 1100 may include discrete components (eg, a full GPU dedicated to computing device 1100 ), virtual components (eg, a portion of a GPU dedicated to computing device 1100 ), or a combination thereof.

[0084] although Fig.11 The various blocks of are shown as being connected with wires via interconnect system 1102, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1118 (such as a display device) may be considered to be I / O component 1114 (e.g., if the display is a touch screen). As another example, CPU 1106 and / or GPU 1108 may include memory (e.g., memory 1104 may represent a storage device in addition to the memory of GPU 1108, CPU 1106, and / or other components). In other words, Fig.11 The computing devices described herein are merely illustrative. No distinction is made between categories such as "workstations," "servers," "laptops," "desktop computers," "tablet computers," "client devices," "mobile devices," "handheld devices," "game consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are contemplated herein. Fig.11 within the range of computing devices.

[0085] The interconnection system 1102 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1102 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, the CPU 1106 may be directly connected to the memory 1104. Further, the CPU 1106 may be directly connected to the GPU 1108. In the case where there is a direct connection or a point-to-point connection between components, the interconnection system 1102 may include a PCIe link to perform the connection. In these examples, the PCI bus does not need to be included in the computing device 1100.

[0086] Memory 1104 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 1100. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0087] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1104 may store computer readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 1100. As used herein, computer storage media does not include the signals themselves.

[0088] Computer storage media may embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal (such as a carrier wave or other transmission mechanism), and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer readable media.

[0089] The CPU 1106 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. Each of the CPUs 1106 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. The CPU 1106 may include any type of processor, and may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers) depending on the type of computing device 1100 implemented. For example, depending on the type of computing device 1100, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as mathematical coprocessors, the computing device 1100 may also include one or more CPUs 1106.

[0090] In addition to or in lieu of the CPU 1106, the GPU(s) 1108 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 1108 may be an integrated GPU (e.g., with one or more of the CPUs 1106) and / or one or more of the GPUs 1108 may be a discrete GPU. In embodiments, one or more of the GPU(s) 1108 may be a coprocessor to one or more of the CPU(s) 1106. The GPU 1108 may be used by the computing device 1100 to render graphics (e.g., 3D graphics) or perform general purpose computing. For example, the GPU 1108 may be used for general purpose computing on a GPU (GPGPU). The GPU 1108 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 1108 may generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command received from the CPU 1106 via a host interface). The GPU 1108 may include a graphics memory, such as a display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 1104. The GPU 1108 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 1108 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

[0091] In addition to or in lieu of the CPU 1106 and / or the GPU 1108, the (one or more) logic units 1120 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. In embodiments, the (one or more) CPUs 1106, the (one or more) GPUs 1108, and / or the (one or more) logic units 1120 may perform any combination of methods, processes, and / or portions thereof, either separately or in conjunction. One or more of the logic units 1120 may be part of and / or integrated into one or more of the CPU 1106 and / or the GPU 1108, and / or one or more of the logic units 1120 may be discrete components or otherwise external to the CPU 1106 and / or the GPU 1108. In embodiments, one or more of logic units 1120 may be coprocessors for one or more of CPU(s) 1106 and / or one or more of GPU(s) 1108 .

[0092] Examples of logic unit 1120 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0093] The communication interface 1110 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1100 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 1110 may include components and functions that enable communication over any of a number of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., via Ethernet or Infiniband communications), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, the logic unit 1120 and / or the communication interface 1110 may include one or more data processing units (DPUs) to transmit data received over the network and / or through the interconnect system 1102 directly to one or more GPUs 1108 (e.g., memory of one or more GPUs 1108).

[0094] I / O ports 1112 can enable computing device 1100 to be logically coupled to other devices including I / O components 1114, presentation components 1118, and / or other components, some of which can be built into (e.g., integrated into) computing device 1100. Illustrative I / O components 1114 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1114 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, the input can be transmitted to an appropriate network element for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1100 (as described in more detail below). The computing device 1100 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1100 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables detection of motion. In some examples, the computing device 1100 may use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0095] The power supply 1116 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1116 may provide power to the computing device 1100 to enable the components of the computing device 1100 to operate.

[0096] One or more presentation components 1118 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 1118 may receive data from other components (e.g., GPU 1108, CPU 1106, DPU, etc.) and output data (e.g., as images, video, sound, etc.).

[0097] Sample Data Center

[0098] Fig.12 An example data center 1200 is shown that can be used in at least one embodiment of the present disclosure. The data center 1200 can include a data center infrastructure layer 1210, a framework layer 1220, a software layer 1230, and / or an application layer 1240.

[0099] like Fig.12 As shown, the data center infrastructure layer 1210 may include a resource coordinator 1212, grouped computing resources 1214, and node computing resources ("node CRs") 1216(1)-1216(N), where "N" represents any integer, positive integer. In at least one embodiment, the node CRs 1216(1)-1216(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more of the node CRs 1216(1)-1216(N) may correspond to a server having one or more of the above-mentioned computing resources. Furthermore, in some embodiments, node CRs 1216 ( 1 )- 1216 1 (N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of node CRs 1216 ( 1 )- 1216 (N) may correspond to a virtual machine (VM).

[0100] In at least one embodiment, the grouped computing resources 1214 may include individual groups of node CRs 1216 housed in one or more racks (not shown) or in many racks in data centers (also not shown) at different geographical locations. Individual groups of node CRs 1216 within the grouped computing resources 1214 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 1216 including CPUs, GPUs, DPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0101] Resource coordinator 1212 may configure or otherwise control one or more node CRs 1216(1)-1216(N) and / or grouped computing resources 1214. In at least one embodiment, resource coordinator 1212 may comprise a software design infrastructure ("SDI") management entity for data center 1200. Resource coordinator 1212 may comprise hardware, software, or some combination thereof.

[0102] In at least one embodiment, Fig.12 As shown, the framework layer 1220 may include a job scheduler 1228, a configuration manager 1234, a resource manager 1236, and / or a distributed file system 1238. The framework layer 1220 may include a framework that supports software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. The software 1232 or the application 1242 may include network-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1220 may be, but is not limited to, a type of free and open source software web application framework that can utilize the distributed file system 1238 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1228 may include a Spark driver to facilitate the scheduling of workloads supported by the various layers of the data center 1200. The configuration manager 1234 may be able to configure different layers, such as the software layer 1230 and the framework layer 1220 including Spark and a distributed file system 1238 for supporting large-scale data processing. The resource manager 1236 may be able to manage clustered or grouped computing resources mapped to or allocated to support the distributed file system 1238 and the job scheduler 1228. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 1214 at the data center infrastructure layer 1210. The resource manager 1236 may coordinate with the resource coordinator 1212 to manage these mapped or allocated computing resources.

[0103] In at least one embodiment, software 1232 included in software layer 1230 may include software used by at least portions of node CRs 1216(1)-1216(N), grouped computing resources 1214, and / or distributed file system 1238 of framework layer 1220. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0104] In at least one embodiment, the applications 1242 included in the application layer 1240 may include one or more types of applications used by the node CRs 1216(1)-1216(N), the grouped computing resources 1214, and / or at least multiple portions of the distributed file system 1238 of the framework layer 1220. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments).

[0105] In at least one embodiment, any of configuration manager 1234, resource manager 1236, and resource coordinator 1212 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. The self-modification actions can save a data center operator of data center 1200 from making potentially poor configuration decisions and potentially avoiding underutilized and / or underperforming portions of the data center.

[0106] According to one or more embodiments described herein, data center 1200 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, (one or more) machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 1200. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1200 by using weight parameters calculated by one or more training techniques (such as, but not limited to, those described herein).

[0107] In at least one embodiment, the data center 1200 may use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or reasoning using the above resources. In addition, the above one or more software and / or hardware resources may be configured to allow users to train or perform information reasoning services, such as image recognition, speech recognition, or other artificial intelligence services.

[0108] Example network environment

[0109] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to communicate with the client device, server, or other device types. Fig.11 The computing device 1100 of the embodiment of the present invention may be implemented on one or more instances of the computing device 1100, for example, each device may include similar components, features and / or functions of the computing device 1100. In addition, in the case of implementing a backend device (e.g., a server, NAS, etc.), the backend device may be included as part of the data center 1200, and the example of the data center 1200 is referred to in this article with respect to Fig.12 Describe in more detail.

[0110] The components of the network environment can communicate with each other via one or more networks that can be wired, wireless, or both. The network can include multiple networks or networks of networks. As an example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or a public switched telephone network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0111] Compatible network environments may include one or more peer-to-peer network environments, in which case the network environment may not include a server, and one or more client-server network environments, in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to (one or more) servers may be implemented on any number of client devices.

[0112] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for one or more applications of the software supporting the software layer and / or the application layer. The software or application may include network-based service software or applications, respectively. In an embodiment, one or more of the client devices may use web-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, the type of free and open source software web application framework, such as a distributed file system that may be used for large-scale data processing (e.g., "big data").

[0113] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed in multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server can assign at least a portion of the function to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0114] One or more client devices may include herein Fig.11At least some of the components, features, and functionality of one or more example computing devices 1100 are described. By way of example and not limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smart phone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a vessel, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0115] The present disclosure may be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) executed by a computer or other machine, such as a personal data assistant or other handheld device. In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs a particular task or implements a particular abstract data type. The present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0116] As used herein, the description of "and / or" about two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. In addition, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0117] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in conjunction with other current or future technologies to include different steps or combinations of steps similar to the steps described in this document. In addition, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, unless and in addition to the order of the individual steps being explicitly described, the terms should not be interpreted as implying any particular order among or between the various steps disclosed herein.

Claims

1. A method comprising: Obtaining first data corresponding to a first three-dimensional 3D face and one or more first feature point positions corresponding to the first 3D face; obtaining second data corresponding to a second 3D face; determining a correspondence between the first 3D face and the second 3D face based at least on performing at least one of one or more transformation processes associated with the first 3D face or one or more fitting processes associated with the first 3D face; determining one or more second feature point positions associated with the second 3D face based at least on the correspondence and using the one or more first feature point positions; as well as Based at least on the one or more second feature point locations, one or more animation operations are performed on the second 3D face.

2. The method according to claim 1, wherein: Performing at least one of the one or more transformation processes associated with the first 3D face or the one or more fitting processes associated with the first 3D face includes: performing a transformation process of the one or more transformation processes by at least updating at least one of a rotation, a translation, or a scale associated with the first 3D face; and A fitting process of the one or more fitting processes is performed by at least updating a shape of the first 3D face.

3. The method according to claim 2, wherein: Executing the fitting process occurs after executing the transform process, and wherein executing at least one of the one or more transform processes associated with the first 3D face or the one or more fitting processes associated with the first 3D face further comprises: After performing the fitting process, performing a second transformation process of the one or more transformation processes by at least further updating at least one of a rotation, a translation, or a scale associated with the first 3D face; and After performing the second transformation process, a second fitting process is performed by at least further updating the shape of the first 3D face.

4. The method according to claim 1, further comprising: receiving input data indicating that a third feature point location associated with the first 3D face corresponds to a fourth feature point location associated with the second 3D face, Wherein determining the corresponding relationship is further based at least on the input data.

5. The method according to claim 1, further comprising: receiving input data indicating that one or more third feature point locations associated with the first 3D face correspond to one or more fourth feature point locations associated with the second 3D face; and updating one or more points of the first 3D face associated with the one or more third feature point positions based at least on the one or more third feature point positions corresponding to the one or more fourth feature point positions, Wherein determining the corresponding relationship is further based at least on updating the one or more points.

6. The method according to claim 1, further comprising: determining a first orientation associated with the first 3D face; as well as determining, based at least on the first orientation, a second orientation associated with the second 3D face such that the second 3D face is substantially oriented with respect to the first 3D face, Wherein determining the corresponding relationship is further based on at least the first orientation and the second orientation.

7. The method according to claim 1, further comprising: determining a first bounding shape associated with the first 3D face; determining a second bounding shape associated with the second 3D face; as well as aligning the first 3D face with respect to the second 3D face based at least on the first enclosing shape and the second enclosing shape, Wherein determining the corresponding relationship is further based at least on aligning the first 3D shape with respect to the second 3D shape.

8. The method according to claim 1, wherein: Determining the one or more second feature point positions associated with the second 3D face comprises: determining potential feature point positions associated with the second 3D face based on at least the correspondence and using a first feature point position of the one or more first feature point positions; determining a first surface normal angle associated with the first feature point position and a second surface normal position associated with the potential feature point position; determining that the first surface normal angle is within a threshold angle of the second surface normal angle; and Based at least on the first surface normal angle being within the threshold angle of the second surface normal angle, determining that the potential feature point location includes a second feature point location among the one or more second feature point locations.

9. The method according to claim 1, wherein: Performing at least one of the one or more transformation processes associated with the first 3D face or the one or more fitting processes associated with the first 3D face uses at least one of: one or more distances between one or more first points associated with the first 3D face and one or more second points associated with the second 3D face; or One or more first surface normal angles associated with the one or more first points and one or more second surface normal angles associated with the one or more second points.

10. The method of claim 1, wherein: The one or more first feature point locations include one or more first locations of one or more facial features associated with the first 3D face; and The one or more second feature point locations include one or more second locations of one or more facial features associated with the second 3D face.

11. A system comprising: One or more processing units, the one or more processing units being configured to: determining a correspondence between the first 3D face and a second 3D face based at least on performing at least one of one or more transformation processes associated with the first three-dimensional 3D face or one or more fitting processes associated with the first 3D face; determining one or more first positions of one or more features associated with the first 3D face; and Based at least on the correspondence and the one or more first positions, one or more second positions of the one or more features associated with the second 3D face are determined.

12. The system according to claim 11, wherein: Performing at least one of the one or more transformation processes associated with the first 3D face or the one or more fitting processes associated with the first 3D face includes: performing a transformation process of the one or more transformation processes by at least updating at least one of a rotation, a translation, or a scale associated with the first 3D face; and A fitting process of the one or more fitting processes is performed by updating at least a shape of the first 3D face.

13. The system according to claim 12, wherein: Executing the fitting process occurs after executing the transformation process, and wherein executing at least one of the one or more transformation processes associated with the first 3D face or the one or more fitting processes associated with the first 3D face further comprises: After performing the fitting process, performing a second transformation process of the one or more transformation processes by at least further updating at least one of a rotation, a translation, or a scale associated with the first 3D face; and After performing the second transformation process, a second fitting process is performed by at least further updating the shape of the first 3D face.

14. The system according to claim 11, wherein: The one or more processing units are further configured to: receiving input data indicating that a third position of a second feature associated with the first 3D face corresponds to a fourth position of the second feature associated with the second 3D face, The corresponding relationship is further determined based on at least the input data.

15. The system according to claim 11, wherein: The one or more processing units are further configured to: receiving input data indicating that one or more third positions of one or more second features associated with the first 3D face correspond to one or more fourth positions of the one or more second features associated with the second 3D face; as well as updating one or more points associated with the one or more second features on the first 3D face to determine one or more updated points based at least on the one or more third locations corresponding to the one or more fourth locations, The corresponding relationship is further determined based at least on the one or more updated points.

16. The system according to claim 11, wherein: The one or more processors are further configured to: determining a first orientation associated with the first 3D face; and determining a second orientation associated with the second 3D face based at least on the first orientation such that the second 3D face is oriented substantially with respect to the first 3D face, The corresponding relationship is further determined based on at least the first orientation and the second orientation.

17. The system of claim 11, wherein: Determining the one or more second positions of the one or more features associated with the second 3D face comprises: determining a potential position of a feature of the one or more features associated with the second 3D face based on at least the correspondence and using a first position of the one or more first positions; determining a first surface normal angle associated with the first position and a second surface normal position associated with the potential position; determining that the first surface normal angle is within a threshold angle of the second surface normal angle; and Based at least on the first surface normal angle being within the threshold angle of the second surface normal angle, determining that the potential position includes a second position of the one or more second positions of the feature.

18. The system of claim 11, wherein: The system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; A system implementing one or more large language models (LLMs); Systems for performing generative AI operations; Systems for generating synthetic data; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A processor comprising: One or more processing units, the one or more processing units being configured to determine one or more first feature point locations associated with a first 3D face using one or more second feature point locations associated with a second three-dimensional 3D face, the second 3D face being processed to be aligned with the first 3D face, wherein the second 3D face is processed to be aligned with the first 3D face based at least on performing one or more transformation processes and one or more fitting processes.

20. The processor of claim 19, wherein the processor is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; A system implementing one or more large language models (LLMs); Systems for generating synthetic data; Systems for performing generative AI operations; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images