Animation redirection using spatial relationships

Abstract

The invention discloses animation redirection using spatial relationships. In various examples, animation redirection using spatial relationships in digital body systems and applications is described herein. Systems and methods for determining constraints using a first point (e.g., a first vertex) associated with a joint and / or grid of a source character and a second point (e.g., a second vertex) associated with a joint and / or grid of a target character are disclosed. As described herein, the constraints may include, but are not limited to, one or more of a deformation constraint, an interaction constraint, a foot constraint, and an angular constraint. Further disclosed are systems and methods that use these constraints in performing animation redirection optimization of a target character. In some examples, the optimization is performed in vertex space, relative to a joint (e.g., rotation and / or transformation of the joint), and / or using one or more techniques (e.g., gradient descent).

Description

Background Art

[0001] Many applications (e.g., gaming applications, interactive applications, communication applications, multimedia applications, etc.) use animated characters or digital avatars to interact with users of the application and / or other animated characters within the application. Therefore, animation retargeting techniques have been developed to make the animations (e.g., movements, actions, etc.) of a target character similar to a source character that can be controlled, animated, and / or authored. For example, some conventional techniques assume that the skeleton of the source character and the skeleton of the target character include the same hierarchy, where the characters are represented using primitive shapes and / or meshes. The skeleton hierarchy can then be used to animate the target character based on the motion of the source character. However, these conventional techniques may not be sufficient to support arbitrary meshes and / or joint hierarchies associated with the character. In addition, other conventional methods use quadratic programming to optimize the character, which may require a large amount of computing resources when performing animation retargeting.

[0002] Therefore, other conventional techniques operate at the mesh level by directly controlling the surface of the character (e.g., the positions of vertices on the mesh). To do this accurately, these conventional techniques require an exact correspondence between points on the source character mesh and points on the target character mesh. However, these conventional techniques do not take into account the skeleton (e.g., joints) of the source character and / or the skeleton (e.g., joints) of the target character when performing the optimization, which is not ideal because common 3D animation pipelines work in terms of joint transforms rather than surface points. Therefore, artists with these techniques must project the solution back into the joint transform space. Furthermore, these conventional techniques need to process all vertices on the character mesh, which again can require significant computational resources when performing animation retargeting. Summary of the Invention

[0003] Embodiments of the present disclosure relate to using spatial relationships for animation retargeting in digital avatar systems and applications. Disclosed are systems and methods for determining constraints using a first point (e.g., a first vertex position vector) associated with a joint and / or mesh of a source character (e.g., a point and / or vertex position vector sampled on the mesh) and a second point (e.g., a second vertex position vector) associated with a joint and / or mesh of a target character (e.g., a point and / or vertex position vector sampled on the mesh). As described herein, the constraints may include, but are not limited to, one or more of a deformation constraint, an interaction constraint, a foot constraint, and / or an angle constraint. Further disclosed are systems and methods for then using these constraints when performing animation retargeting optimization for the target character. In some examples, the optimization is performed for the joints (e.g., rotations and / or transformations of the joints), such as in joint transformation space, and / or in vertex space, and / or may be performed using one or more other techniques (e.g., gradient descent).

[0004] Compared to conventional systems, the systems and methods described herein are capable, in some embodiments, of performing optimization using points on joints and points on a character's mesh. This provides for more realistic animation retargeting (e.g., target character animation) compared to conventional systems that perform optimization using only the character's joint hierarchy and conventional systems that perform optimization using only the vertices of the character's mesh. Furthermore, compared to conventional systems, in some embodiments, the systems perform optimization in vertex space, joint transform space, and / or using specific techniques (e.g., gradient descent and / or more advanced modifications (e.g., Hessian optimization, etc.), which may result in the current system performing with less latency and / or requiring fewer computational resources than conventional systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present system and method for using spatial relationships for animation retargeting in digital or virtual avatar / human systems and applications is described in detail below with reference to the accompanying drawings, wherein:

[0006] Figure 1 An example of a process for performing animation retargeting by optimizing various constraints according to some embodiments of the present disclosure is shown;

[0007] Figure 2 shows an example illustration of a character that may be used for animation retargeting according to some embodiments of the present disclosure;

[0008] Figures 3A to 3B shows an example of correspondence between points associated with a source character and points associated with a target character according to some embodiments of the present disclosure;

[0009] Figure 4 shows examples of deformation constraints associated with animation retargeting according to some embodiments of the present disclosure;

[0010] Figure 5 shows examples of interaction constraints associated with animation redirection according to some embodiments of the present disclosure;

[0011] Figure 6 shows an example of acceleration constraints associated with animation redirection according to some embodiments of the present disclosure;

[0012] 7A to 7B shows an example of a foot constraint associated with animation redirection according to some embodiments of the present disclosure;

[0013] Figure 8 shows examples of joint constraints associated with animation retargeting according to some embodiments of the present disclosure;

[0014] Figure 9A flowchart of a method for performing animation retargeting using spatial information according to some embodiments of the present disclosure is shown;

[0015] Figure 10 A flowchart illustrating a method of determining that a point associated with a source character corresponds to a point associated with a target character according to some embodiments of the present disclosure is shown;

[0016] Figure 11 A flowchart illustrating a method for performing animation retargeting between a character and an object using spatial information according to some embodiments of the present disclosure is shown;

[0017] Figure 12 is a block diagram of an exemplary content streaming system suitable for implementing some embodiments of the present disclosure;

[0018] Figure 13 is a block diagram of an exemplary computing device suitable for implementing some embodiments of the present disclosure; and

[0019] Figure 14 is a block diagram of an exemplary data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Systems and methods are disclosed relating to animation retargeting using spatial information in digital or virtual avatar / human systems and applications. For example, a system may receive first data representing a source character and second data representing a target character. As described herein, the source character may include a character that is performing an animation (e.g., a movement), and the target character may include a character that is replicating (e.g., mimicking) the animation of the source character. In some examples, the target character may include one or more characteristics that are similar to one or more characteristics of the source character. Additionally, or alternatively, in some examples, the target character may include one or more characteristics that are different from one or more characteristics of the source character. As described herein, characteristics may include, but are not limited to, character type, height, weight, number of joints, joint positions, joint types, and / or any other characteristics that may be used to create a character.

[0021] The system can then determine points associated with the source character that correspond to points associated with the target character. To determine the correspondence between these points, the system can first determine one or more first points associated with one or more joints of the source character and one or more second points associated with one or more corresponding joints of the target character. In some examples, the system uses input (e.g., tags) from one or more users to determine the one or more first points associated with the one or more joints and / or the one or more second points associated with the one or more corresponding joints, while in other examples, the system determines the one or more first points associated with the one or more joints and / or the one or more second points associated with the one or more corresponding joints based at least on processing of the character. The system can then use the one or more first points and the one or more second points to determine one or more third points associated with the first mesh of the source character, which correspond to one or more fourth points associated with the second mesh of the target character. Thus, the system can use a smaller subset of points to create an initial association between the two characters, and can then use the smaller subset to match additional (e.g., remaining) points.

[0022] For example, to determine a correspondence between points, the system can determine a first vector between a first point associated with a joint of the source character and a third point located on the first mesh of the source character. The system can also determine one or more second vectors between a second point associated with a corresponding joint of the target character and one or more fourth points located on the second mesh of the target character. The system can then use the second vectors and the first vectors to determine that one of the fourth points corresponds to the third point. For example, the system can determine that the fourth point that includes the vector closest to the first vector corresponds to the third point. The system can then continue to perform these processes for any number of points associated with the source character and / or any number of points associated with the target character.

[0023] The system can then use the correspondences to determine constraints, which can be preserved when performing animation retargeting optimization. In some examples, the constraints can include one or more deformation constraints associated with spatial information between points on the character. For example, the system can identify a first set of points on the source character that meets one or more thresholds, such as one or more distance thresholds (described in more detail herein). The system can then identify a second set of points on the target character that corresponds to the first set of points. In addition, the system can determine the deformation constraints based on at least a first vertex position and / or a first vector associated with the first set of points and a second vertex position and / or a second vector associated with the second set of points. For example, the deformation constraint may be associated with making the second vector similar to the first vector. The system may then use a similar process for other sets of points on the source character and / or the target character.

[0024] In some examples, the constraints may include one or more interaction constraints associated with the relationship between the character and the object. For example, the system may identify a first set of points between the source character and the object, where the first set of points satisfies one or more thresholds, such as one or more distance thresholds (described in more detail herein). The system may then identify a second set of points between the target character and the corresponding object, where the second set of points corresponds to the first set of points. In addition, the system may determine the interaction constraints based at least on a first vertex position and / or a first vector associated with the first set of points and a second vertex position and / or a second vector associated with the second set of points. For example, the interaction constraint may be associated with making the second vector similar to the first vector. The system may then use a similar process for additional sets of points between the character and the object.

[0025] In some examples, the constraints may include one or more acceleration constraints associated with the target character. For example, the system may determine the acceleration constraint based on the vertex positions associated with one or more points associated with the target character at multiple time instances (e.g., between multiple frames). In some examples, the constraints may include one or more joint constraints associated with angles between pairs of joints. For example, the system may determine the joint constraint based on at least a first vertex position and / or a first vector between a first set of points associated with two joints of the source character and / or a second vertex position and / or a second vector between a second set of points associated with the same two joints of the target character. For example, the joint constraint may be associated with making the second vector similar to the first vector (e.g., including an angle similar to the first vector). The system may then use a similar process for additional point sets associated with additional joints of the source character and / or target character.

[0026] In some examples, the constraints can include one or more foot or bipedal constraints associated with the character. For the first example, the foot constraint can be based at least on causing one or more vertex positions of one or more points associated with a first foot of the target character to be substantially similar to one or more vertex positions of one or more corresponding points associated with a corresponding foot of the source character and / or at least on causing a second foot of the target character to include an orientation (e.g., in the y-direction) substantially similar to the corresponding foot of the source character. For the second example, the foot constraint can be based at least on vertex positions of one or more points associated with one or more feet of the target character at multiple time instances (e.g., between multiple frames), such as when the corresponding foot of the source character remains substantially stationary between the multiple time instances. Although primarily described herein as feet, the constraints can be applied to other features, such as fins, hooves, flippers, etc., of any type of character / avatar / etc.

[0027] The system may then perform the optimization using one or more formulas that use one or more constraints described herein. In some examples, the formula may include a similar form of one or more (e.g., each) constraint used for optimization. In some examples, the formula may optimize for one or more variables, such as variables associated with one or more degrees of freedom of one or more joints corresponding to the target character. As described herein, the degrees of freedom associated with a joint may be associated with a translation and / or rotation associated with the joint. In some examples, the formula may be associated with one or more optimization techniques, such as gradient descent. For example, gradient descent may be used to perform the optimization for the variables.

[0028] As described herein, the system may perform optimization for one or more offline applications and / or one or more online applications. In some examples, the system may perform optimization for offline applications by executing these formulations for multiple frames (e.g., every frame) in a single instance. In some examples, the system may perform optimization for online applications by executing the formulations for one or more frames at different time points as the system continues to generate new frames. In addition, in some examples, such as when executing an online application, the formulation used for optimization may include one or more additional terms. For example, the formulation may include one or more regularization terms to improve the motion associated with the target character between frames, such as reducing artifacts and / or jitter. In addition, the optimization may be performed at each frame without requiring any information about the motion from future frames, and / or the optimization may not depend on the positions of future vertices. The system may then cause output associated with the source character and / or the target character. For example, the output may include the poses of the target character at different frames (e.g., the target character's joint transforms matching those of the source character), where the frames are associated with different time instances. For example, the output may include a video of the target character animated similarly to the source character.

[0029] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein may be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational artificial intelligence, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, gaming, 3D content (e.g., gaming, entertainment, media, etc.), cloud computing, and / or any other suitable application.

[0030] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems for performing conversational AI operations, systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0031] refer to Figure 1 , Figure 1An example of a process 100 for performing animation redirection by optimizing various constraints according to some embodiments of the present disclosure is shown. It should be understood that this arrangement and other arrangements described herein are presented as examples only. In addition to the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in a memory.

[0032] Process 100 may include an association component 102 receiving source character data 104 representing a source character and target character data 106 representing a target character. As described herein, a source character may include a character / avatar / digital human / etc. that is performing animations (e.g., movements), and a target character may include a character that is replicating (e.g., mimicking) the animations of the source character. In some examples, the target character may include one or more characteristics that are similar to one or more characteristics of the source character. Additionally, or alternatively, in some examples, the target character may include one or more characteristics that are different from one or more characteristics of the source character. As described herein, characteristics may include, but are not limited to, character type, height, weight, number of joints, joint positions, joint types, and / or any other characteristics that may be used to create a character.

[0033] For example, Figure 2 An example diagram of characters that may be used for animation redirection according to some embodiments of the present disclosure is shown. As shown, animation redirection may include using a source character 202 (which may be represented by source character data 104) and a target character 204 (which may be represented by target character data 106). Figure 2 The example depicts target character 204 as including similar characteristics as source character 202, such as similar parts (e.g., a head, two arms, a torso, two legs, etc.), a similar number of joints, and / or similar joint positions, but in other examples, the target character can include different characteristics compared to the source character. For example, in other examples, the target character can include different parts, a different number of joints, different joint positions, etc.

[0034] review Figure 1For example, process 100 may include an association component 102 determining points associated with a source character that correspond to points associated with a target character. In some examples, association component 102 may use any technique to determine points associated with the source character that correspond to points associated with the target character. Additionally, as described herein, corresponding points may be associated with joints of a character, a mesh of a character, an object with which a character interacts, and / or any other points.

[0035] For an example of how the association component 102 determines correspondences between points, the association component 102 can first determine one or more first points associated with one or more joints of the source character and one or more second points associated with one or more corresponding joints of the target character. In some examples, the system uses input (e.g., labels) from one or more users to determine the first point associated with the joint and / or the second point associated with the corresponding joint, while in other examples, the system determines the first point associated with the joint and / or the second point associated with the corresponding joint based on at least the processing character. In some examples, the labels can then be used to transfer the correspondences between the joints to the correspondences between the mesh surfaces.

[0036] For example, as shown, the association component 102 can generate user interface data 108 representing a user interface (e.g., similar to Figure 2 ). One or more user devices 110 can then present a user interface to one or more users using the user interface data 108. The user can then provide input using the user device 110, which can be represented by input data 112, wherein the input adds a label to the joint of the source character and adds a similar label to the corresponding joint of the target character. For example, if the joint is associated with the right elbow, the user can provide a label of "right elbow" for the source character and another label of "right elbow" for the target character. In response, the association component 102 can cause the points associated with the joint on the source character to correspond to the points of the corresponding joint on the target character. This process can then be repeated for any number of joints associated with the source character and / or the target character.

[0037] In some examples, the user can use any technology when providing input. For a first example, the user device 110 may include input devices, such as a keyboard, a joystick, a mouse, buttons, a touch display, etc., which the user can use to provide input. For a second example, the user device 110 may include one or more microphones that the user device 110 uses to generate audio data representing speech associated with the input (e.g., a voice command). In such an example, the user device 110 may use one or more machine learning models, one or more neural networks, one or more large language models (LLMs), and / or any other components to process the audio data in order to interpret the command as input.

[0038] The association component 102 can then use the first and second points to determine one or more third points on the source character's mesh that correspond to one or more fourth points on the target character's mesh. For example, to determine the correspondence between the points, the association component 102 can determine a first vector between a first point associated with a joint of the source character and a third point located on the source character's mesh. The association component 102 can also determine one or more second vectors between a second point associated with a corresponding joint of the target character and one or more fourth points located on the target character's mesh. The association component 102 can then use the second and first vectors to determine that one of the fourth points corresponds to the third point. For example, the association component 102 can determine that the fourth point that includes the vector closest to the first vector corresponds to the third point. The association component 102 can then continue to perform these processes for any number of points associated with the source character and / or any number of points associated with the target character.

[0039] For example, Figures 3A to 3B 2 shows an example of correspondence between points associated with the source character 202 and points associated with the target character 204 according to some embodiments of the present disclosure. Figure 3A As shown in the example of , a user may first mark a first point 302 associated with a joint of the source character 202, where the first point 302 is gray because it is located on the skeleton of the source character 202. The user may then mark a second point 304 associated with a corresponding joint on the target character 204, where the second point 304 is also gray because it is located on the skeleton of the target character 204. Based on these inputs, the association component 102 may determine that the first point 302 corresponds to the second point 304, which is indicated by line 306.

[0040] The association component 102 can then determine a first vector 308 between a third point 310 on the mesh of the source character 202 and the first point 302 (e.g., the nearest joint point), where the third point 310 is black because it is located on the mesh of the source character 202. Additionally, the association component 102 can determine at least a second vector 312 between a fourth point 314 on the mesh of the target character 204 and the second point 304, and a third vector 316 between a fifth point 318 on the mesh of the target character 204 and the second point 304, where points 314 and 318 are also black because they are located on the mesh of the target character 204. The association component 102 can then use the first vector 308, the second vector 312, and the third vector 316 to determine whether the fourth point 314 or the fifth point 318 corresponds to the third point 310. For example, in Figure 3A In the example, the association component 102 can determine that the fourth point 314 corresponds to the third point 310 based at least on the second vector 312 being more similar to the first vector 308 than the third vector 316 being similar to the first vector 308 , where the correspondence is indicated by line 320 .

[0041] This process can then be repeated for any number of points associated with the joints and / or mesh of the source character 202 and / or any number of points associated with the joints and / or mesh of the target character 204. For example, Figure 3B The example further illustrates that the association component 102 determines that points 322(1)-322(7) associated with the joints of the source character 202 correspond to points 324(1)-324(7), respectively, associated with the corresponding joints of the target character 204. Additionally, the association component 102 determines that points 326(1)-326(3) associated with the mesh of the source character 202 correspond to points 328(1)-328(3), respectively, associated with the mesh of the target character 204. Figure 3B The examples shown in FIG. 2 show a particular number of points, but in other examples, the association component 102 can perform these processes to determine that any number of points associated with the source character 202 corresponds to any number of points associated with the target character 204. For example, the association component 102 can determine a correspondence between one point, ten points, one hundred points, one thousand points, and / or any other number of points.

[0042] As described herein, while this is merely one example technique of how the association component 102 can determine associations between points of the source character 202 and points of the target character 204 , in other examples, the association component 102 can use any other technique.

[0043] review Figure 1For example, process 100 may include an association component 102 generating and / or outputting point data 114. Point data 114 may represent positions (e.g., vertex position vectors) of points associated with a skeleton and / or mesh of a source character, labels of points associated with the source character, positions (e.g., vertex position vectors) of points associated with a skeleton and / or mesh of a target character, labels of points associated with the target character, correspondences between points associated with the source character and points associated with the target character, and / or any other information associated with points of the character. Process 100 may then include an optimization component 116 using at least point data 114 to perform one or more optimization techniques described herein to perform animation retargeting for the target character.

[0044] As described herein, the optimization component 116 can determine constraints that can be preserved when performing optimization for animation retargeting. For example, in some examples, the constraints can include one or more deformation constraints 118 associated with spatial information between points on the character. For example, the optimization component 116 can identify a first set of points on the source character that satisfies one or more thresholds 120 (e.g., one or more distance thresholds 120). For example, to identify the first set of points on the source character, the optimization component 116 can use the vertex position of the first point and the vertex position of the second point to determine a distance d between the points. The optimization component 116 can then determine whether the first point and the second point should be included in the first set of points based at least on whether the distance d satisfies (e.g., is less than) the following relationship:

[0045] d<ε (1)

[0046] d / d0<γ (2)

[0047] In formulas (1) and (2), ε and γ comprise the threshold distance 120, and d0 comprises the distance between the same two points when the source character and / or the target character are in a set pose (e.g., a default pose and / or an initial pose), e.g. Figures 3A to 3B . In some examples, the threshold distance ε is similar to the threshold distance γ, while in other examples, the threshold distance ε is different from the threshold distance γ. Furthermore, in some examples, the optimization component 116 can determine that the point should be included in the first point set based at least on satisfying one of the threshold distances, while in other examples, the optimization component 116 can determine that the point should be included in the first point set based at least on satisfying both threshold distances.

[0048] The optimization component 116 can then identify a second set of points on the target character that corresponds to the first set of points (e.g., using the correspondence determined by the correspondence component 102). Furthermore, the optimization component 116 can determine a deformation constraint 118 based on at least the first vertex position and / or first vector associated with the first set of points and the second vertex position and / or second vector associated with the second set of points. For example, the deformation constraint 118 can be associated with making the second vector similar to the first vector using the following formula:

[0049]

[0050] In formula (3), and is the vertex position of the point contained in the second point set, is the vector between the points in the first point set. As described herein, vector positions may include (V x ,V y ,V z ). The optimization component 116 can then use a similar process to identify one or more additional point sets on the source character and / or the target character.

[0051] For example, Figure 4 An example of a deformation constraint associated with animation redirection according to some embodiments of the present disclosure is shown. As shown, the optimization component 116 can perform one or more processes described herein to identify a first set of points including a first point 402 and a second point 404 associated with the source character 202. The optimization component 116 can then perform one or more processes described herein to identify a second set of points including a third point 406 and a fourth point 408 associated with the target character 204, wherein the third point 406 corresponds to the first point 402 and the fourth point 408 corresponds to the second point 404. The deformation constraint can then be associated with maintaining distances between the second set of points similar to distances between the first set of points and / or causing vectors between the second set of points to include the same direction as vectors between the first set of points. In other words, the deformation constraint can cause the vectors between the second set of points to be similar to the vectors between the first set of points. Additionally, a similar process can be performed for additional sets of points associated with the source character 202 and / or additional sets of points associated with the target character 204, as determined by Figure 4 The line in the figure shows the .

[0052] review Figure 1For example, the constraints may include one or more interaction constraints 122 associated with spatial information between a point associated with the character and a point associated with an object with which the character interacts. For example, the optimization component 116 may identify a first point set that includes a first point associated with the source character and a second point associated with the object, wherein the first point set is identified based at least on the first point set satisfying one or more distance thresholds 120. For example, to identify the first point set, the optimization component 116 may use the vertex position of the first point associated with the source character and the vertex position of the second point associated with the object to determine a distance d between the points. i The optimization component 116 can then be configured to calculate the distance d based on at least i Whether the first point and the second point should be included in the first point set is determined by whether the following conditions are met (e.g., less than):

[0053] d i <β (4)

[0054] In equation (4), β comprises the threshold distance 120. In some examples, the threshold distance β is similar to one or more of the threshold distance ε or the threshold distance γ, while in other examples, the threshold distance β is different from both the threshold distance ε and the threshold distance γ.

[0055] Then, the optimization component 116 can identify a second point set that includes a third point associated with the target character and a fourth point associated with the corresponding object, wherein the second point set corresponds to the first point set. In addition, the optimization component 116 can determine the interaction constraint 122 based on at least the first vertex position and / or first vector associated with the first point set and the second vertex position and / or second vector associated with the second point set. For example, the interaction constraint 122 can be associated with making the second vector similar to the first vector using the following formula:

[0056]

[0057] In formula (5), and is the vertex position of the point contained in the second point set, is the vector between the points in the first point set.The optimization component 116 can then use a similar process to identify one or more additional point sets associated with characters and objects.

[0058] For example, Figure 51 shows an example of interaction constraints associated with animation redirection according to certain embodiments of the present disclosure. As shown, the optimization component 116 can perform one or more processes described herein to identify a first set of points, the first set of points including a first point 502 associated with the source character 202 and a second point 504 associated with the first object 506. The optimization component 116 can also identify a second set of points including a third point 508 associated with the target character 204 and a fourth point 510 associated with the second object 512. Figure 5 In the example of , third point 508 corresponds to first point 502, fourth point 510 corresponds to second point 504, and second object 512 corresponds to first object 506. The interaction constraint can then be associated with maintaining the distance between the second set of points similar to the distance between the first set of points, and / or with causing the vector between the second set of points to include the same direction as the vector between the first set of points. In other words, the interaction constraint can cause the vector between the second set of points to be similar to the vector between the first set of points. In addition, a similar process can be performed for one or more additional point sets associated with source character 202 and first object 506 and / or one or more additional point sets associated with target character 204 and second object 512.

[0059] review Figure 1 In an example, the constraints may include one or more acceleration constraints 124 corresponding to one or more accelerations of one or more points associated with the target character (and / or, in some examples, the source character). For example, for a point associated with the target character, the optimization component 116 may determine the vertex position of the point at different time instances. In some examples, one or more time instances (e.g., each time instance) may correspond to a corresponding frame depicting the target character. For example, for the current frame, the optimization component 116 may determine a first vertex position of the point in the previous frame, a second vertex position of the point in the current frame, and a third vertex position of the point in the future frame. The optimization component 116 may then use the vertex positions to determine the acceleration constraints 124. For example, the acceleration constraints 124 may be associated with the following equation:

[0060]

[0061] In formula (6), is the vertex position of the point at the previous time instance (e.g., the previous frame), is the vertex position of the point at the current time instance (e.g., the current frame), is the vertex position of the point at a future time instance (e.g., a future frame). The optimization component 116 can then use a similar process for one or more additional points associated with the target character. In some examples, acceleration constraints 124 can be used to control the acceleration of different points and / or parts of the target character, for example, to avoid jitter between frames.

[0062] For example, Figure 6 6 shows an example of an acceleration constraint associated with animation redirection according to some embodiments of the present disclosure. As shown, over a time period 602, the optimization component 116 may determine a first vertex position 604 of a point at a first time instance T(1) (e.g., ), the second vertex position 606 of the point at the second time instance T(2) (eg, ) and a third vertex position 608 of the point at the third time instance T(3) (e.g., ). In some examples, the first time instance T(1) is associated with the previous frame, the second time instance T(2) is associated with the current frame, and the third time instance T(3) is associated with the future frame. The optimization component 116 can then use the vertex positions 604, 606, and 608 as part of the acceleration constraint 124 associated with the target character 204 (e.g., using equation (6)). Furthermore, in some examples, the optimization component 116 can perform a similar process for one or more additional points associated with the target character 204.

[0063] review Figure 1 , the constraints may include one or more single-foot or dual-foot constraints 126 associated with the motion and / or position of one or more feet of the target character. For example, in some examples, a first foot constraint 126 may include causing one or more points associated with a foot of the target character to include one or more identical positions of one or more corresponding points associated with the same foot of the source character. For example, for a point associated with a foot of the source character, the optimization component 116 may determine a first vertex position of the point associated with the source character and a second vertex position of the corresponding point associated with the target character. For the foot constraint 126, the optimization component 116 may then use the following formula:

[0064]

[0065] In formula (7), is the first vertex position of the point associated with the source actor, and is the second vertex position of the corresponding point associated with the target character. As shown, formula (7) makes the second vertex position similar to the first vertex position. The optimization component 116 can then use a similar process for one or more additional points associated with the source character and / or one or more additional points associated with the target character.

[0066] In some examples, the second foot constraint 126 may include causing the target character's other foot to have the same position as the source character's corresponding foot on one or more axes. As described herein, the orientation may be relative to the x-direction, the y-direction, and / or the z-direction. For example, the optimization component 116 may cause the y-direction coordinate of the first point associated with the source character's foot to be similar to the y-direction coordinate of the point associated with the target character's corresponding foot. For example, the foot constraint 126 may be associated with the following formula:

[0067]

[0068] In formula (8), is the first vertex position of the point associated with the source actor, and is the second vertex position of the corresponding point associated with the target character. The optimization component 116 can then use a similar process for one or more additional points associated with the source character and / or one or more additional points associated with the target character. Thus, by using these foot constraints 126, the target character's feet can be oriented and / or animated similarly to the source character's feet.

[0069] For example, Figure 7A 1 shows an example of a foot constraint associated with animation redirection according to some embodiments of the present disclosure. As shown, the optimization component 116 can determine a first vertex position 702 of a point associated with the source character 202 (e.g., ) and the second vertex position 704 of the corresponding point associated with the target character 204 (e.g., ). The optimization component 116 can then use the vertex positions 702 and 704 as part of the foot constraint 126, which causes the feet of the target character 204 to be located approximately in the same position as the corresponding feet of the source character 202 (e.g., using equation (7)). The optimization component 116 can also determine a third vertex position 706 of a point associated with the source character 202 (e.g., ) and a fourth vertex position 708 of a corresponding point associated with the target character 204 (e.g., ). The optimization component 116 can then use the vertex positions 706 and 708 as part of the foot constraint 126, which causes the feet of the target character 204 to include substantially the same orientation as the corresponding feet of the source character 202 (eg, using equation (8)).

[0070] review Figure 1For example, in some examples, the third foot constraint 126 may include limiting the movement (e.g., sliding) of one or more feet of the target character. For example, the optimization component 116 may determine that the foot of the source character is stationary and / or substantially stationary between two time instances (e.g., between two frames). In some examples, the optimization component 116 may determine that the foot is stationary and / or substantially stationary based at least on one or more differences between one or more first vertex positions of one or more points associated with the foot at the first time instance and one or more second vertex positions of one or more points at the second time instance satisfying (e.g., being less than) a threshold. If the optimization component 116 determines that the foot is stationary and / or substantially stationary, the optimization component 116 may limit the movement of the corresponding foot of the target character.

[0071] For example, the optimization component 116 can determine a first vertex position of a point associated with a corresponding foot of the target character at a first time instance and a second vertex position of a point associated with the corresponding foot at a second time instance. The optimization component 116 can then determine the foot constraint 126 using the following formula:

[0072]

[0073] In formula (9), is the vertex position of the point at the previous time instance (e.g., the previous frame), and is the vertex position of the point at the current time instance (e.g., the current frame). The optimization component 116 can then use a similar process for one or more other points associated with the target character. In some examples, the foot constraint 126 can be used to control the movement of one or more feet of the target character, such as to avoid "sliding" between frames. In some examples, the foot constraint 126 can be used based at least on the occurrence of one or more other factors, such as when the speed associated with the corresponding foot of the source character is less than a threshold speed. In other words, the foot constraint 126 can be configured to keep one or more feet of the target character stationary when one or more corresponding feet of the source character are stationary.

[0074] For example, Figure 7B , another example of a foot constraint related to animation redirection according to some embodiments of the present disclosure is shown. As shown, during a period of time 710, the optimization component 116 may determine a first vertex position 712 of a point at a first time instance T(1) (e.g., ) and the second vertex position 714 of the point at the second time instance T(2) (e.g., ). In some examples, the first time instance T(1) is associated with the previous frame, and the second time instance T(2) is associated with the current time. The optimization component 116 can then use the vertex positions 712 and 714 as part of the foot constraint 126 associated with the target character 204 (e.g., using formula (9)). Furthermore, in some examples, the optimization component 116 can perform a similar process for one or more additional points associated with the target character 204.

[0075] review Figure 1 In an example of, the constraints may include one or more joint constraints 128 associated with maintaining an angle between one or more joint sets associated with a source character and one or more corresponding joint sets associated with a target character. For example, the optimization component 116 may determine a first set of points associated with the joints of the source character and a second set of points associated with the corresponding joints of the target character. The optimization component 116 may then determine the joint constraints 128 based on at least a first vertex position and / or a first vector associated with the first set of points and a second vertex position and / or a second vector associated with the second set of points. As described herein, the joint constraints 128 may be associated with making a first angle associated with the first vector similar and / or substantially similar to a second angle associated with the second vector. For example, the optimization component 116 may determine the joint constraints 128 using the following formula:

[0076]

[0077] In formula (10), is the vertex position of the point associated with the first joint of the source character (e.g., the first label), is the vertex position of the point (e.g., the second label) associated with the corresponding first joint of the target character, is the vertex position of the point associated with the second joint of the source character (e.g., the corresponding first label), and is the vertex position of the point associated with the corresponding second joint of the target character (e.g., the corresponding second label). For example, the first label may include the "right shoulder" of the source character, the second label may include the "right elbow" of the source character, the corresponding first label may also include the "right shoulder" of the target character, and the corresponding second label may also include the "right elbow" of the target character. The optimization component 116 can then use a similar process for additional point sets on the source character and / or the target character.

[0078] For example, Figure 8 1 shows an example of joint constraints associated with animation redirection according to some embodiments of the present disclosure. As shown, the optimization component 116 can determine a first point set including first vertex positions 802 of points associated with a first joint of the source character 202 (e.g., ) and a second vertex position 804 of a point associated with a second joint of the source character 202 (e.g., ). In some examples, the optimization component 116 can identify the joints as comprising a pair of joints, wherein no other joints are located between the first joint and the second joint. The optimization component 116 can then determine a second set of points comprising third vertex positions 806 of points associated with corresponding first joints of the target character 204 (e.g., ) and a fourth vertex position of a point associated with a corresponding second joint of the target character 204 (e.g., ). The optimization component 116 can then use the vertex positions 802, 804, 806, and 808 as part of the joint constraints 128 associated with the source character 202 and the target character 204 (e.g., using equation (10)). Furthermore, in some examples, the optimization component 116 can perform a similar process for one or more pairs of joints associated with the source character 202 and / or the target character 204.

[0079] review Figure 1 In an example of a target character, process 100 may include an optimization component 116 using one or more optimization techniques 130 to optimize animation retargeting, for example, based on one or more constraints. In some examples, as described herein, optimization component 116 may perform optimization for one or more joints of a source character and / or a target character. For example, optimization component 116 may perform optimization based on at least one or more degrees of freedom associated with a joint of a target character. As described herein, a degree of freedom may be for a rotation associated with a joint, a translation associated with a joint, and / or a position associated with a joint. For example, optimization technique 130 may include using the following formula:

[0080]

[0081] As shown, Equation (11) includes a general form associated with one or more (e.g., each) of the constraints described herein. For example, V may include a vertex position vector (V ) associated with a point located on a joint of a mesh or character. x ,V y ,V z ), α can be associated with a set of points, where α includes a first value (e.g., -1) for the first point in the set and a second value (e.g., 1) for the second point in the set, p iA target value (e.g., a value associated with a source character described herein) may be included, and θ may include a rotation and translation (and / or, in some examples, a position) of the joint. In some examples, θ may be a change (e.g., a delta transform) with respect to a transformation, such as from a previous pose associated with the target character. In some examples, θ may be a transformation (e.g., a delta transform) with respect to a redirection solution obtained using one or more additional and / or alternative algorithms. For example, the additional algorithm may ignore mesh information, such that the additional algorithm works at a skeleton level to compute an initial estimate of θ. A final θ may then be determined based at least on the delta transform from the initial estimate of θ, where θ represents the final delta transform.

[0082] Furthermore, in some examples, rotations and / or translations (and / or, in some examples, positions) can be constrained. For example, translations associated with one or more joint scalings can be frozen, such as scalings of all joints except the root joint. While formula (11) may include only a general form of the optimization, the optimization can be performed using one or more other optimization formulas described herein (e.g., using one or more constraints described herein). For example, the optimization can be performed by summing one or more constraints described herein.

[0083] In some examples, the optimization component 116 can perform the optimization using gradient descent, such as by:

[0084]

[0085] In some examples, the optimization component 116 can use one or more additional operations when performing the optimization. For a first example, the optimization component 116 can assume that one or more joints of the source character include one or more vertex positions that are the same as one or more corresponding joints of the target character. For example, the optimization component 116 can assume that the hip joint of the source character includes the same vertex position as the hip joint of the target character (e.g., the angle connecting these joints is perpendicular). For a second example, the optimization component 116 can lock one or more joints, such as by locking the rotation, transformation, and / or position of the joint. For a third example, the optimization component 116 can provide one or more weights for one or more constraints described herein.

[0086] In addition, as described herein, the optimization component 116 can perform optimizations for both offline and online applications. In some examples, the optimization component 116 can perform optimizations for offline applications by performing these formulations on multiple frames (e.g., each frame) in a single instance. For example, offline applications may mean that processing is performed asynchronously and / or not in real time. In some examples, the optimization component 116 can perform optimizations for online applications by performing formulations on one or more frames at different time instances as the optimization component 116 continues to receive new frames for performing animation redirection. For example, online applications may mean that processing is performed synchronously and / or in near real time and / or in real time. Therefore, the optimization component 116 can perform one or more additional processes when performing animation redirection for online applications.

[0087] For example, the optimization component 116 can add one or more adjustment terms, such as the following:

[0088]

[0089] In formula (13), the adjustment term can limit θ at the current time instance (e.g., the current frame), such as a rotation associated with a joint, such as by using gradient descent from a previous time instance (e.g., the previous frame). Thus, in some examples, the solution for the current time instance can be in absolute value form and / or can be in incremental form with respect to the previous time instance. Additionally, or alternatively, in some examples, θ can be a transformation (e.g., an incremental transformation) with respect to a redirected solution obtained using one or more additional and / or alternative algorithms. For example, the additional algorithm can ignore mesh information such that the additional algorithm operates at the skeleton level to compute an initial estimate of θ. A final θ can then be determined based at least on the incremental transformation from the initial estimate of θ, where θ represents the final incremental transformation. For example, the final θ can be determined using the initial estimate and / or one or more formulas described herein (e.g., formula (13)).

[0090] For example, in the following formula, the q term can be an incremental transformation applied to the solution from a simple solution without considering the grid. In addition, one or more adjustment terms can be defined as follows:

[0091]

[0092] In formulas (14) and (15), q j is the quaternion of joint j, is the quaternion of joint j at the previous frame t-1, is the quaternion of joint j at the current frame t. In other words, the q term can be in delta form as discussed further in this paper.

[0093] Process 100 may include the optimization component 116 generating and / or outputting optimization data 132 representing the animation redirection associated with the target character. For example, the optimization data 132 may represent a pose associated with the target character at various frames. In some examples, the pose may be associated with a vertex position of various points associated with the target character. In some examples, a computing device (e.g., user device 110) may then use the optimization data 132 to display the animation redirection associated with the target character.

[0094] Now refer to Figures 9 to 11 , each block of methods 900, 1000, and 1100 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. Methods 900, 1000, and 1100 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 900, 1000, and 1100 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Furthermore, methods 900, 1000, and 1100 are described by way of example with respect to Figure 1 However, these methods 900, 1000, and 1100 may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0095] Figure 9 A flow chart of a method 900 for animation retargeting using spatial information, according to some embodiments of the present disclosure, is shown. Method 900, at block B902, may include determining that a first point associated with a source character corresponds to a second point associated with a target character. For example, association component 102 may determine that a first point associated with a source character corresponds to a second point associated with a target character. As described herein, association component 102 may use one or more techniques (e.g., user input and / or character processing) to determine that the first point corresponds to the second point.

[0096] Method 900, at block B904, can include determining one or more first point sets from the first point. For example, optimization component 116 can determine the first point set. As described herein, in some examples, optimization component 116 can determine the first point set based at least on determining that distances between points included in the first point set satisfy (e.g., are less than) one or more threshold distances 120. In other words, optimization component 116 can identify the first point set as including points that are in close proximity to one another.

[0097] Method 900 may include determining one or more first vectors associated with one or more first point sets at block B906. For example, optimization component 116 may determine the first vectors associated with the first point set. In some examples, optimization component 116 determines the first vectors using vertex positions of first points included in the first point set.

[0098] Method 900 may include, at block B908, determining a pose associated with the target character based at least on corresponding one or more second vectors associated with one or more second point sets in the second points to one or more first vectors. For example, the optimization component 116 may determine the pose associated with the target character by corresponding the second vectors associated with the second point set to the first vector associated with the first point set. As described herein, the correspondence may be associated with one or more deformation constraints 118. Additionally, in some examples, the optimization component 116 may also use additional constraints (e.g., interaction constraints 122, acceleration constraints 124, foot constraints 126, joint constraints 128, etc.) to determine the pose.

[0099] At block B910, method 900 may include causing the presentation of a target character based at least on the pose. For example, optimization component 116 may generate and / or output optimization data 132 representing at least the pose. A computing device (e.g., user device 110) may then use optimization data 132 to present the target character in the pose. Furthermore, in some examples, method 900 may continue to repeat to determine additional poses for additional frames associated with the target character.

[0100] Figure 10 A flow chart of a method 1000 for determining that a point associated with a source character corresponds to a point associated with a target character, according to some embodiments of the present disclosure, is shown. Method 1000, at block B1002, may include determining that a first point associated with a joint of the source character corresponds to a second point of the corresponding joint of the target character. For example, association component 102 may determine that a first point associated with a joint of the source character corresponds to a second point of the corresponding joint of the target character. In some examples, association component 102 may make this determination based at least on receiving input indicating similar labels for the joints.

[0101] Method 1000 may include, at block B1004, determining a first vector between a third point associated with the first mesh of the source character and the first point. For example, association component 102 may determine the first vector between the third point associated with the first mesh of the source character and the first point. In some examples, association component 102 makes this determination based at least on a vertex position associated with the first point and a vertex position associated with the third point.

[0102] Method 1000 may include, at block B1006, determining one or more second vectors between one or more fourth points associated with the second mesh of the target character and the second point. For example, association component 102 may determine the second vector between the fourth point associated with the second mesh of the target character and the second point. In some examples, association component 102 makes this determination based at least on a vertex position associated with the second point and one or more vertex positions associated with the fourth point.

[0103] Method 1000 may include, at block B1008, determining that a fourth point among the one or more fourth points corresponds to the third point based at least on the first vector and the one or more second vectors. For example, association component 102 may determine that the fourth point corresponds to the third point based at least on the first vector and the second vector. In some examples, association component 102 makes this determination based at least on the fourth point being associated with the vector of the second vectors that is closest to the first vector.

[0104] Figure 11 A flow chart of a method 1100 for performing animation retargeting between a character and an object using spatial information according to some embodiments of the present disclosure is shown. The method 1100 may include, at block B1102, determining a first point set that includes at least a first point associated with a source character and a second point associated with the object. For example, the optimization component 116 may determine the first point set that includes at least a first point associated with the source character and a second point associated with the object. In some examples, the optimization component 116 may determine the first point set based at least on determining that a distance between points included in the first point set satisfies (e.g., is less than) one or more threshold distances 120. In other words, the optimization component 116 may identify the first point set as including points that are in close proximity to each other.

[0105] Method 1100 may include, at block B1104, determining a second point set that includes at least a third point associated with the target character and a fourth point associated with the corresponding object. For example, optimization component 116 may determine the second point set that includes at least the third point associated with the target character and the fourth point associated with the corresponding object. In some examples, optimization component 116 may determine the second point set based at least on the third point corresponding to the first point and the fourth point corresponding to the second point.

[0106] Method 1100 may include, at block B1106, determining a pose between the target character and the corresponding object based at least on the correspondence between the first point set and the second point set. For example, optimization component 116 may determine the pose based at least on the correspondence between the first point set and the second point set. In some examples, optimization component 116 performs the correspondence by optimizing a first vector associated with the first point set with a second vector associated with the second point set. In some examples, optimization component 116 performs the optimization in joint space, as described herein.

[0107] At block B1108, method 1100 may include causing the presentation of the target character based on at least the gesture. For example, optimization component 116 may generate and / or output optimization data 132 representing at least the gesture. A computing device (e.g., user device 110) may then use optimization data 132 to present the target character in the gesture. Additionally, in some examples, method 1100 may continue to repeat to determine additional gestures for additional frames associated with the target character.

[0108] Example content streaming system

[0109] Now see Figure 12 , Figure 12 is an example system diagram for a content streaming system 1200 according to some embodiments of the present disclosure. Figure 12 Includes one or more application servers 1202 (which may include Figure 13 ), one or more client devices 1204 (which may include components, features, and / or functionality similar to the example computing device 1300 of Figure 13 1300) and one or more networks 1206 (which may be similar to one or more networks described herein). In some embodiments of the present disclosure, the system 1200 may be implemented. The application sessions may correspond to game streaming applications (e.g., NVIDIA GeFORCE NOW), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulation), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.

[0110] In system 1200, for an application session, one or more client devices 1204 may receive only input data in response to input to one or more input devices, transmit the input data to one or more application servers 1202, receive encoded display data from the one or more application servers 1202, and display the display data on a display 1224. Thus, more computationally intensive calculations and processing are offloaded to one or more application servers 1202 (e.g., rendering for graphical output of the application session—particularly ray or path tracing—is performed by one or more GPUs of one or more game servers 1202). In other words, the application session is streamed from one or more application servers 1202 to one or more client devices 1204, thereby reducing the graphics processing and rendering requirements of one or more client devices 1204.

[0111] For example, with respect to instantiation of an application session, client device 1204 can display a frame of the application session on display 1224 based on receiving display data from one or more application servers 1202. Client device 1204 can receive input from one of the one or more input devices and generate input data in response. Client device 1204 can send the input data to application server 1202 via communication interface 1220 and via network 1206 (e.g., the Internet), and application server 1202 can receive the input data via communication interface 1218. The CPU can receive the input data, process the input data, and transmit the data to the GPU, which causes the GPU to generate a rendering of the application session. For example, the input data can represent movement of a user's character in a game session of a gaming application, firing a weapon, reloading, passing a ball, steering a vehicle, and the like. Rendering component 1212 can render the application session (e.g., representing the results of the input data), and rendering capture component 1214 can capture the rendering of the application session as display data (e.g., as image data capturing a rendered frame of the application session). Rendering of the application session may include ray or path-traced lighting and / or shading effects computed using one or more parallel processing units (such as GPUs) of application server(s) 1202, which may further utilize one or more specialized hardware accelerators or processing cores to perform the ray or path tracing techniques. In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components such as vGPUs, vCPUs, etc.—may be used by application server 1202 to support the application session. Encoder 1216 may then encode the display data to generate encoded display data, and the encoded display data may be sent to client device 1204 via communication interface 1218 over network 1206. Client device 1204 may receive the encoded display data via communication interface 1220, and decoder 1222 may decode the encoded display data to generate display data. Client device 1204 may then display the display data via display 1224.

[0112] The systems and methods described herein may be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environment or object simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0113] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems that perform deep learning operations, systems that perform simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems that include one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems for performing conversational AI operations, systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0114] Example computing device

[0115] Figure 13 13 is a block diagram of an example computing device 1300 suitable for implementing some embodiments of the present disclosure. Computing device 1300 may include an interconnect system 1302 that directly or indirectly couples the following devices: memory 1304, one or more central processing units (CPUs) 1306, one or more graphics processing units (GPUs) 1308, a communication interface 1310, input / output (I / O) ports 1312, I / O components 1314, a power supply 1316, one or more presentation components 1318 (e.g., one or more displays), and one or more logic units 1320. In at least one embodiment, one or more computing devices 1300 may include one or more virtual machines (VMs), and / or any of their components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of GPUs 1308 may include one or more vGPUs, one or more of CPUs 1306 may include one or more vCPUs, and / or one or more of logic units 1320 may include one or more virtual logic units. As such, one or more computing devices 1300 may include discrete components (eg, a full GPU dedicated to computing device 1300 ), virtual components (eg, a portion of a GPU dedicated to computing device 1300 ), or a combination thereof.

[0116] although Figure 13The various blocks of are shown as being connected with lines via interconnect system 1302, but this is not intended to be limiting and is provided merely for clarity. For example, in some embodiments, presentation component 1318 (such as a display device) may be considered to be I / O component 1314 (e.g., if the display is a touch screen). As another example, CPU 1306 and / or GPU 1308 may include memory (e.g., memory 1304 may represent a storage device in addition to the memory of GPU 1308, CPU 1306, and / or other components). In other words, Figure 13 The computing devices referred to herein are illustrative only. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are contemplated. Figure 13 within the range of computing devices.

[0117] Interconnect system 1302 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. Interconnect system 1302 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 1306 can be directly connected to memory 1304. Further, CPU 1306 can be directly connected to GPU 1308. In the case where there is a direct connection or a point-to-point connection between components, interconnect system 1302 can include a PCIe link to perform the connection. In these examples, it is not necessary to include a PCI bus in computing device 1300.

[0118] Memory 1304 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1300. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0119] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1304 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 1300. As used herein, computer storage media does not include signals themselves.

[0120] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal (such as a carrier wave or other transport mechanism), and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0121] The CPU 1306 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1300 to perform one or more of the methods and / or processes described herein. Each of the CPUs 1306 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. The CPU 1306 may include any type of processor and may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers) depending on the type of computing device 1300 implemented. For example, depending on the type of computing device 1300, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, the computing device 1300 may also include one or more CPUs 1306.

[0122] In addition to or in lieu of CPU 1306, one or more GPUs 1308 may be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1300 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1308 may be integrated GPUs (e.g., with one or more of CPUs 1306) and / or one or more of GPUs 1308 may be discrete GPUs. In embodiments, one or more of one or more GPUs 1308 may be a coprocessor for one or more of one or more of CPUs 1306. GPU 1308 may be used by computing device 1300 to render graphics (e.g., 3D graphics) or perform general-purpose computations. For example, GPU 1308 may be used for general-purpose computing on a GPU (GPGPU). GPU 1308 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 1308 may generate pixel data for output images in response to rendering commands (e.g., rendering commands received from CPU 1306 via a host interface). GPU 1308 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1304. GPU 808 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 1308 may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

[0123] In addition to or in lieu of the CPU 1306 and / or GPU 1308, one or more logic units 1320 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1300 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 1306, the one or more GPUs 1308, and / or the one or more logic units 1320 may independently or jointly perform any combination of methods, processes, and / or portions thereof. One or more of the logic units 1320 may be a part of and / or integrated within one or more of the CPUs 1306 and / or GPUs 1308, and / or one or more of the logic units 1320 may be discrete components or otherwise external to the CPUs 1306 and / or GPUs 1308. In embodiments, one or more of the logic units 1320 may be coprocessors for one or more of the one or more CPUs 1306 and / or one or more of the one or more GPUs 1308.

[0124] Examples of logic unit 1320 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0125] The communication interface 1310 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1300 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 1310 may include components and functionality that enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., via Ethernet or InfiniBand communication), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1320 and / or the communication interface 1310 may include one or more data processing units (DPUs) to transmit data received over the network and / or through the interconnect system 1302 directly to one or more GPUs 1308 (e.g., memory of one or more GPUs 1308).

[0126] I / O ports 1312 can enable computing device 1300 to be logically coupled to other devices including I / O components 1314, presentation components 1318, and / or other components, some of which can be built into (e.g., integrated into) computing device 1300. Illustrative I / O components 1314 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. I / O components 1314 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some instances, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1300 (as described in more detail below). Computing device 1300 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, or a combination thereof, for gesture detection and recognition. Additionally, computing device 1300 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) to enable motion detection. In some examples, computing device 1300 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0127] The power supply 1316 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1316 may provide power to the computing device 1300 to enable the components of the computing device 1300 to operate.

[0128] One or more presentation components 1318 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. Presentation component 1318 may receive data from other components (e.g., GPU 1308, CPU 1306, DPU, etc.) and output data (e.g., as images, video, sound, etc.).

[0129] Sample Data Center

[0130] Figure 14 An example data center 1400 that can be used in at least one embodiment of the present disclosure is shown. The data center 1400 can include a data center infrastructure layer 1410, a framework layer 1420, a software layer 1430, and / or an application layer 1440.

[0131] like Figure 14 As shown, the data center infrastructure layer 1410 may include a resource coordinator 1412, grouped computing resources 1414, and node computing resources ("node CRs") 1416(1)-1416(N), where "N" represents any integer, positive integer. In at least one embodiment, the node CRs 1416(1)-1416(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output ("NW"), and other network devices. I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more of the node CRs 1416(1)-1416(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node CRs 1416(1)-1416(N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of the node CRs 1416(1)-1416(N) may correspond to virtual machines (VMs).

[0132] In at least one embodiment, the grouped computing resources 1414 may include separate groups of node CR1416 housed in one or more racks (not shown) or in many racks housed in data centers (also not shown) at different geographical locations. The separate groups of node CR1416 within the grouped computing resources 1414 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CR1416 including CPUs, GPUs, DPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0133] Resource coordinator 1412 may configure or otherwise control one or more nodes CR 1416(1)-1416(N) and / or grouped computing resources 1414. In at least one embodiment, resource coordinator 1412 may comprise a software design infrastructure ("SDI") management entity for data center 1400. Resource coordinator 1412 may comprise hardware, software, or some combination thereof.

[0134] In at least one embodiment, Figure 14 As shown, the framework layer 1420 may include a job scheduler 1428, a configuration manager 1434, a resource manager 1436, and / or a distributed file system 1438. The framework layer 1420 may include a framework that supports software 1432 of the software layer 1430 and / or one or more applications 1442 of the application layer 1440. The software 1432 or the application 1442 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1420 may be, but is not limited to, a free and open source software web application framework type such as Apache Spark that can utilize the distributed file system 1438 for large-scale data processing (e.g., "big data"). TM(hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 1428 may include a Spark driver to facilitate the scheduling of workloads supported by the various layers of data center 1400. Configuration manager 1434 may be capable of configuring different layers, such as software layer 1430 and framework layer 1420 including Spark and distributed file system 1438 for supporting large-scale data processing. Resource manager 1436 may be capable of managing clustered or grouped computing resources mapped to or allocated to support distributed file system 1438 and job scheduler 1428. In at least one embodiment, clustered or grouped computing resources may include grouped computing resources 1414 at data center infrastructure layer 1410. Resource manager 1436 may coordinate with resource coordinator 1412 to manage these mapped or allocated computing resources.

[0135] In at least one embodiment, the software 1432 included in the software layer 1430 may include software used by at least portions of the node CRs 1416(1)-1416(N), the grouped computing resources 1414, and / or the distributed file system 1438 of the framework layer 1420. The one or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0136] In at least one embodiment, the applications 1442 included in the application layer 1440 may include one or more types of applications used by the nodes CRs 1416(1)-1416(N), the grouped computing resources 1414, and / or at least portions of the distributed file system 1438 of the framework layer 1420. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments).

[0137] In at least one embodiment, any of configuration manager 1434, resource manager 1436, and resource coordinator 1412 can implement any number and type of self-modification actions based on any number and type of data obtained in any technically feasible manner. The self-modification actions can save the data center operator of data center 1400 from making potentially poor configuration decisions and potentially avoiding underutilized and / or underperforming portions of the data center.

[0138] In accordance with one or more embodiments described herein, data center 1400 can include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by computing weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 1400. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1400 using weight parameters computed using one or more training techniques, such as, but not limited to, those described herein.

[0139] In at least one embodiment, data center 1400 may use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or reasoning using the aforementioned resources. In addition, the aforementioned one or more software and / or hardware resources may be configured to allow users to train or perform information reasoning services, such as image recognition, speech recognition, or other artificial intelligence services.

[0140] Sample network environment

[0141] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to: Figure 13 1300, for example, each device may include similar components, features, and / or functions of the computing device 1300. In addition, in the case of implementing a backend device (e.g., a server, NAS, etc.), the backend device may be included as part of the data center 1400, and the example of the data center 1400 is referred to herein with respect to Figure 14 Describe in more detail.

[0142] The components of the network environment can communicate with each other via one or more networks that can be wired, wireless, or both. The network can include multiple networks or networks of networks. As an example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0143] Compatible network environments may include one or more peer-to-peer network environments, in which case the network environment may not include a server, and one or more client-server network environments, in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0144] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include software supporting the software layer and / or a framework for one or more applications of the application layer. The software or application may include network-based service software or applications, respectively. In an embodiment, one or more of the client devices may use web-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework, such as one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0145] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to the user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0146] One or more client devices may include herein Figure 13At least some of the components, features, and functionality of one or more example computing devices 1300 are described. By way of example and not limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a vessel, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of the depicted devices, or any other suitable device.

[0147] The present disclosure may be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, and the like, refer to code that performs a particular task or implements a particular abstract data type. The present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, and the like. The present disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices linked through a communications network.

[0148] As used herein, the statement "and / or" with respect to two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0149] The subject matter of the present disclosure is described herein with specificity to satisfy statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may also be embodied in other ways, in conjunction with other current or future technologies, to include different steps or combinations of steps similar to the steps described in this document. Furthermore, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, such terms should not be construed to imply any particular order among or between the various steps disclosed herein unless and except where the order of the individual steps is explicitly described.

[0150] Sample Clauses

[0151] A: A method comprising: determining a first plurality of points associated with a source character, the first plurality of points corresponding to a second plurality of points associated with a target character; determining one or more first point sets from the first plurality of points; determining one or more first vectors associated with the one or more first point sets; determining a pose associated with the target character based at least on adjusting one or more second vectors associated with one or more second point sets in the second plurality of points to correspond to the one or more first vectors; and causing the target character to be presented based at least on the pose.

[0152] B: A method as described in paragraph A, wherein determining a point set from the one or more first point sets includes: determining a distance between a first point and a second point in the point set; and determining that the distance is less than a threshold distance.

[0153] C: A method as described in paragraph A or paragraph B, wherein determining a point set from the one or more first point sets includes: determining a first distance between a first point and a second point in the point set when the source character is in a current posture; determining a second distance between the first point and the second point when the source character is in a set posture; determining a third distance based on at least the first distance and the second distance; and determining that the third distance is less than a threshold distance.

[0154] D: A method as described in any of paragraphs AC, wherein determining that the first plurality of points corresponds to the second plurality of points includes: determining one or more first points from the first plurality of points associated with one or more joint labels; determining one or more second points from the second plurality of points associated with the one or more joint labels; and determining that the one or more first plurality of points corresponds to the one or more second plurality of points based at least on the one or more joint labels.

[0155] E: A method as described in any of paragraphs AD, wherein determining that a first point in the first plurality of points corresponds to a second point in the second plurality of points includes: determining a first vector between the first point associated with the first mesh of the source character and a third point in the first plurality of points, the third point being associated with a first joint of the source character; determining a second vector between the second point associated with the second mesh of the target character and a fourth point in the second plurality of points, the fourth point being associated with a second joint of the target character, the second joint corresponding to the first joint of the source character; and determining that the first point corresponds to the second point based at least on the first vector and the second vector.

[0156] F: A method as described in any of paragraphs AE, wherein determining the posture associated with the target character includes: performing an optimization operation on one or more degrees of freedom associated with one or more joints of the target character so that the one or more second vectors correspond to the one or more first vectors; and determining the posture associated with the target character based at least on the one or more degrees of freedom determined during the optimization operation.

[0157] G: The method of paragraph F, wherein the optimization operation comprises gradient descent.

[0158] H: The method as described in any of paragraphs AG further includes: determining one or more third point sets between the source character and the first object; determining one or more third vectors associated with the one or more third point sets; and determining one or more fourth point sets between the target character and the second object corresponding to the first object based at least on the one or more third point sets, wherein determining the posture associated with the target character is also based at least on adjusting one or more fourth vectors associated with the one or more fourth point sets to correspond to the one or more third vectors.

[0159] I: The method as described in any of paragraphs AH also includes: determining that a first point of the first plurality of points is static between a first frame and a second frame, wherein the first point is associated with a foot of the source character; determining a first position of a second point of the second plurality of points at the first frame, the second point corresponding to the first point; and determining a second position of the second point at the second frame, wherein determining the posture associated with the target character is also based at least on the first position and the second position.

[0160] J: The method as described in any one of paragraphs AI further includes: determining one or more third point sets from the first plurality of points associated with the joints of the source character; determining one or more third vectors associated with the one or more third point sets; and determining one or more fourth point sets from the second plurality of points corresponding to the one or more third point sets, wherein determining the posture associated with the target character is also based at least on adjusting one or more fourth vectors associated with one or more fourth point sets to correspond to the one or more third vectors.

[0161] K: A system comprising: one or more processors for: determining one or more first point sets associated with a first mesh of a source character; determining one or more second point sets associated with a second mesh of a target character based at least on the one or more first point sets; determining a pose associated with the target character based at least on performing an optimization operation on one or more joints of the target character using the one or more first point sets and the one or more second point sets; and causing the target character to be rendered based at least on the pose.

[0162] L: A system as described in paragraph K, wherein determining a point set from the one or more first point sets includes: determining a distance between a first point and a second point in the point set; and determining that the distance is less than one or more threshold distances.

[0163] M: A system as described in paragraph K or paragraph L, wherein determining a point set from the one or more first point sets includes: determining a first distance between a first point and a second point in the point set when the source character is in a current pose; determining a second distance between the first point and the second point when the source character is in a set pose; determining a third distance based on at least the first distance and the second distance; and determining that the third distance is less than a threshold distance.

[0164] N: A system as described in any of paragraphs KM, wherein determining that the one or more second point sets correspond to the one or more first point sets is based on at least at least one of the following: one or more inputs indicating that a first plurality of points in the one or more first point sets corresponds to a second plurality of points in the one or more second point sets; or determining that one or more first vectors between the first plurality of points and one or more joint points associated with the source character correspond to one or more second vectors between the second plurality of points and one or more corresponding joint points associated with the target character.

[0165] O: A system as described in any of paragraphs KN, wherein determining the posture associated with the target character includes: performing an optimization operation on one or more degrees of freedom associated with the one or more joints of the target character so that one or more first vectors associated with the one or more first point sets correspond to one or more second vectors associated with the one or more second point sets; and determining the posture associated with the target character based at least on the one or more degrees of freedom determined during the optimization operation.

[0166] P: A system as described in any of paragraphs KO, wherein the one or more processors are further used to: determine one or more third point sets between the source character and the first object; and determine one or more fourth point sets between the target character and the second object corresponding to the first object based at least on the one or more third point sets; wherein determining the posture associated with the target character is also based at least on the one or more third point sets and the one or more fourth point sets.

[0167] Q: A system as described in any of paragraphs KP, wherein the one or more processors are further used to: determine that a third point associated with the foot of the source character is static between a previous frame and a current frame; determine a first position of a fourth point associated with the target character in the previous frame based at least on the third point being static, the fourth point corresponding to the third point; and determine a second position of the fourth point in the current frame, wherein determining the posture associated with the target character is also based at least on the first position and the second position.

[0168] R: A system as described in any of paragraphs KQ, wherein the system is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system associated with a gaming application; a system associated with a three-dimensional content application; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

[0169] S: One or more processors, comprising: processing circuitry for generating an animation of a target character in a pose corresponding to the source character based at least on adjusting one or more joints associated with the target character using one or more first point sets associated with a first mesh of the source character and one or more second point sets associated with a second mesh of the target character.

[0170] T: One or more processors as described in paragraph S, wherein the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system associated with a gaming application; a system associated with a three-dimensional content application; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

Claims

1. A method comprising: determining a first plurality of points associated with a source character, the first plurality of points corresponding to a second plurality of points associated with a target character; determining one or more first point sets from the first plurality of points; determining one or more first vectors associated with the one or more first point sets; determining a pose associated with the target character based at least on adjusting one or more second vectors associated with one or more second sets of points in the second plurality of points to correspond to the one or more first vectors; as well as The target character is caused to be rendered based at least on the gesture.

2. The method according to claim 1, wherein Determining a point set from the one or more first point sets includes: determining a distance between a first point and a second point in the point set; and It is determined that the distance is less than a threshold distance.

3. The method according to claim 1, wherein Determining a point set from the one or more first point sets includes: determining a first distance between a first point and a second point in the point set when the source character is in a current pose; determining a second distance between the first point and the second point when the source character is in a set pose; determining a third distance based at least on the first distance and the second distance; and It is determined that the third distance is less than a threshold distance.

4. The method according to claim 1, wherein Determining that the first plurality of points corresponds to the second plurality of points includes: determining one or more first points from the first plurality of points associated with the one or more joint labels; determining one or more second points from the second plurality of points associated with the one or more joint labels; and The one or more first plurality of points is determined to correspond to the one or more second plurality of points based at least on the one or more joint labels.

5. The method according to claim 1, wherein Determining that a first point in the first plurality of points corresponds to a second point in the second points includes: determining a first vector between the first point associated with a first mesh of the source character and a third point of the first plurality of points, the third point associated with a first joint of the source character; determining a second vector between the second point associated with a second mesh of the target character and a fourth point in the second plurality of points, the fourth point being associated with a second joint of the target character, the second joint corresponding to the first joint of the source character; and It is determined based on at least the first vector and the second vector that the first point corresponds to the second point.

6. The method according to claim 1, wherein Determining the posture associated with the target character includes: performing an optimization operation on one or more degrees of freedom associated with one or more joints of the target character such that the one or more second vectors correspond to the one or more first vectors; and The pose associated with the target character is determined based at least on the one or more degrees of freedom determined during the optimization operation. The method of claim 6 , wherein the optimization operation comprises gradient descent.

8. The method according to claim 1, further comprising: determining one or more third point sets between the source character and the first object; determining one or more third vectors associated with the one or more third point sets; as well as determining one or more fourth point sets between the target character and a second object corresponding to the first object based at least on the one or more third point sets, Wherein determining the pose associated with the target character is further based on at least adjusting one or more fourth vectors associated with the one or more fourth point sets to correspond to the one or more third vectors.

9. The method according to claim 1, further comprising: determining a first point of the first plurality of points to be static between a first frame and a second frame, wherein the first point is associated with a foot of the source character; determining a first position of a second point of the second plurality of points at the first frame, the second point corresponding to the first point; as well as determining a second position of the second point at a second frame, Wherein determining the posture associated with the target character is further based on at least the first position and the second position.

10. The method according to claim 1, further comprising: determining one or more third point sets from the first plurality of points associated with joints of the source character; determining one or more third vectors associated with the one or more third point sets; as well as determining one or more fourth point sets from said second plurality of points corresponding to said one or more third point sets, Wherein, determining the pose associated with the target character is further based on at least adjusting one or more fourth vectors associated with one or more fourth point sets to correspond to the one or more third vectors.

11. A system comprising: One or more processors for: determining one or more first point sets associated with a first mesh of a source character; determining one or more second point sets associated with a second mesh of the target character based at least on the one or more first point sets; determining a pose associated with the target character based at least on performing an optimization operation for one or more joints of the target character using the one or more first point sets and the one or more second point sets; as well as The target character is caused to be rendered based at least on the gesture.

12. The system according to claim 11, wherein Determining a point set from the one or more first point sets includes: determining a distance between a first point and a second point in the point set; and It is determined that the distance is less than one or more threshold distances.

13. The system of claim 11 , wherein determining a point set from the one or more first point sets comprises: determining a first distance between a first point and a second point in the point set when the source character is in a current pose; determining a second distance between the first point and the second point when the source character is in a set pose; determining a third distance based at least on the first distance and the second distance; as well as It is determined that the third distance is less than a threshold distance.

14. The system according to claim 11, wherein: Determining that the one or more second point sets correspond to the one or more first point sets is based on at least one of the following: one or more inputs indicating that a first plurality of points in the one or more first point sets corresponds to a second plurality of points in the one or more second point sets; or Determining one or more first vectors between the first plurality of points and one or more joint points associated with the source character corresponds to one or more second vectors between the second plurality of points and one or more corresponding joint points associated with the target character.

15. The system of claim 11 , wherein determining the posture associated with the target character comprises: performing an optimization operation on one or more degrees of freedom associated with the one or more joints of the target character such that one or more first vectors associated with the one or more first point sets correspond to one or more second vectors associated with the one or more second point sets; as well as The pose associated with the target character is determined based at least on the one or more degrees of freedom determined during the optimization operation.

16. The system according to claim 11, wherein The one or more processors are further configured to: determining one or more third point sets between the source character and the first object; as well as determining one or more fourth point sets between the target character and a second object corresponding to the first object based at least on the one or more third point sets; Wherein determining the pose associated with the target character is further based on at least the one or more third point sets and the one or more fourth point sets.

17. The system of claim 11, wherein the one or more processors are further configured to: determining that a third point associated with a foot of the source character is static between a previous frame and a current frame; Based at least on the fact that the third point is static, determining a first position of a fourth point associated with the target character in a previous frame, the fourth point corresponding to the third point; and determining a second position of the fourth point in the current frame, Wherein determining the posture associated with the target character is further based on at least the first position and the second position.

18. The system of claim 11, wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; Systems for performing operations using large language models; A system for performing one or more conversational AI operations; Systems for generating synthetic data; Systems associated with gaming applications; A system associated with a three-dimensional content application; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. One or more processors comprising: Processing circuitry for generating an animation of a target character in a pose corresponding to the source character based at least on adjusting one or more joints associated with the target character using one or more first point sets associated with a first mesh of the source character and one or more second point sets associated with a second mesh of the target character.

20. The one or more processors of claim 19, wherein the one or more processors are included in at least one of: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; Systems for performing operations using large language models; A system for performing one or more conversational AI operations; Systems for generating synthetic data; Systems associated with gaming applications; A system associated with a three-dimensional content application; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images