AUDIO-DRIVEN FACIAL ANIMATION USING MACHINE LEARNING

Abstract

Systems and methods of this disclosure involve animating virtual avatars or agents according to input audio and one or more selected or specific emotions and / or styles. For example, a deep neural network can be trained to output motion or deformation information for a character representing the character speaking speech contained in the audio input. The character may have various facial components or regions (e.g., head, skin, eyes, tongue) that are modeled separately, so that the network can output motion or deformation information for each of these different facial components. During training, the network may use a transformer-based audio encoder with locked parameters to train an associated decoder using a weighted feature vector.The network output can be provided to a renderer to generate audio-driven facial animation that is emotion-accurate.

Description

CROSS-REFERENCE TO THE RELATED REGISTRATION

[0001] This application claims the priority and benefit of Chinese patent application No. 202311036144,8, filed on August 16, 2023, entitled “AUDIO-DRIVED FACIAL ANIMATION USING MACHINE LEARNING”, the full disclosure of which is hereby incorporated by reference in its entirety for all purposes. BACKGROUND

[0002] It may be desirable for various operations used to animate a character to appear as if that character is speaking, represented by audio data. Partly due to the time and complexity involved in creating such animation, automating this process can be advantageous, especially for real-time or near-real-time operations. Approaches based on machine learning have been used to generate character animation based on input audio, but these earlier approaches are generally limited in their capabilities, producing animation that is often not sufficiently realistic.For example, an earlier approach might attempt to animate various facial features of a character, including the mouth or eyes, to correspond to speech represented by appropriate audio data—but these models often fail to produce realistic animations when used in native languages ​​for which the model has not been explicitly trained. This problem can be exacerbated for operations where the character is a virtual human intended to appear as an actual human, speaking language realistically with realistic behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments according to the present disclosure are described with reference to the drawings, in which: Fig. 1A and Fig. 1B Illustrate aspects of a character that can be animated at least partially based on speech data, according to at least one embodiment; Fig. 2. An exemplary network for generating animation is illustrated to match language, including emotion and style support, according to at least one embodiment; Fig. 3 An exemplary network for generating animation is illustrated to correspond to language, according to at least one embodiment; Fig. 4 illustrates an exemplary training process for a network to generate animation of a character to match input audio, according to at least one embodiment; Fig. 5A and Fig. 5B Illustrate exemplary inference processes for a network to generate animation of a character to match input audio, according to at least one embodiment; Fig. Six components of a distributed system are illustrated that can be used to update or perform inference using a machine learning model, according to at least one embodiment; Fig. 7A Inference and / or training logic illustrated, according to at least one embodiment; Fig. 7B Inference and / or training logic illustrated, according to at least one embodiment; Fig. 8 illustrates an exemplary data center system, according to at least one embodiment; Fig. 9 illustrates a computer system according to at least one embodiment; Fig. 10 illustrates a computer system according to at least one embodiment; Fig. 11 at least sections of a graphics processor illustrated, according to one or more embodiments; Fig. 12 at least sections of a graphics processor illustrated, according to one or more embodiments; Fig. 13 an exemplary data flow diagram for an advanced computing pipeline, according to at least one embodiment; Fig. 14 a system diagram for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment; and Fig. 15A and Fig. 15B illustrates a data flow diagram for a process for training a machine learning model and a client-server architecture to improve annotation tools with pre-trained annotation models, according to at least one embodiment. DETAILED DESCRIPTION

[0004] The following description details various embodiments. For illustrative purposes, specific configurations and details are presented to provide a thorough understanding of these embodiments. However, it is also apparent to those skilled in the art that the embodiments can be implemented without these specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiment.

[0005] The systems and procedures described herein may be used by, among others, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in an in-cabin infotainment system or a digital or virtual driver assistance application), autonomous vehicles or machines, controlled and uncontrolled robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled to one or more trailers, flying ships, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other types of vehicles.Furthermore, the systems and methods described herein can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine driving, synthetic data generation, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational artificial intelligence (AI), generative AI with large language models (LLMs), light transport simulation (e.g., ray tracking, path tracking, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable applications.

[0006] Disclosed embodiments may be included in a wide variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, air systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, and systems for performing conversational AI operations.Systems for performing generative AI operations using LLMs, systems for performing light transport simulation, systems for performing collaborative content generation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0007] Approaches according to various embodiments can generate an animation depicting one or more characters speaking speech represented by audio data. This could include, for example, a high-resolution, full three-dimensional (3D) facial animation for presentation as part of a film, a game, an artificial intelligence (AI)-based agent, a digital avatar, a teleconference, a virtual / augmented / mixed / extended reality experience, and / or another media presentation, content, or experience. One or more deep neural networks, such as…A frame-based convolutional neural network (CNN), a recurrent neural network (RNN), and / or a transformer-based model can take raw audio as input, extract features from the raw audio, and receive one or more component vectors to animate a character to utter speech contained in an audio segment extracted from the input raw audio. The network can then provide output, such as motion, vertex, and / or deformation data, which can be provided to a renderer to, for example, generate or synthesize the facial animation corresponding to that part of the speech. During training, the network can receive one or more component vectors, such as...The network receives a style vector or an emotion vector specifying one or more emotions with potential relative weights to be used for rendering facial animation for an input audio clip. The emotion(s) specified by the data can change at various points or keyframes in the audio data. The network can also receive a style vector (or style information contained within the emotion vector or other emotional data representation) specifying a modification or fine-tuning of the animation to be generated for the specified emotion(s). This can refer to an animation style, among other options, or to specific movements to be modified or extended.The motion or deformation information output by the mesh can correspond to a set of facial components (or other body components) or segments that can be animated at least somewhat independently to realistically represent the character speaking input language. These components can include, for example, the character's head, jaw, eyeballs, tongue, or skin. In embodiments, body components or segments—such as arms, legs, torso, neck, etc.—can be modeled in addition to or as an alternative to facial components or segments.Modeling each of these facial (and / or body) components separately and determining deformations for each of these components can make the rendered facial (and / or body) animation appear more realistic for a given emotion, especially when taking into account any style data provided to the network.

[0008] Several systems and methods are designed to incorporate one or more transformer-based audio encoders into an Audio2Face model. Various embodiments make modifications to a formant analysis network of the Audio2Face model to modify existing CNN-based systems with a pre-trained transformer model. In at least one embodiment, modifications involve, at least in part, incorporating one or more transformer layers into an audio encoder model, which may also include one or more CNN layers. The model can be pre-trained on audio data, which may include a variety of different languages, and then used within the Audio2Face model to assist in training a decoder to generate a set of vectors for use by one or more application-specific outputs, such as within a renderer to create an animated face (e.g., a video).to generate a character that replicates how a human face might say the input audio sample. The pre-trained transformer model can include a number of layers, and given an input set of features from CNN layers, each layer can generate a set of vectors associated with those features. Systems and methods add a combination layer, which can include learnable weights that can be applied to linearly combine each of the output vectors for each layer to produce a final output. The combination layer weights can be learned as the model is trained by backpropagation. Various embodiments can be implemented using short-duration audio samples (e.g.,(approximately less than 1 second, approximately less than 0.8 seconds, approximately less than 0.6 seconds, approximately between 0.4 and 0.6 seconds, approximately between 0.6 and 0.8 seconds, approximately between 0.8 and 1 second, or any other reasonable duration) and may not be able to capture information from previous frames. As a result, a user can select any section of a larger audio clip for evaluation and viewing without losing output quality due to a lack of previous frame knowledge. Additionally, techniques for improving output quality can be included. In at least one embodiment, an additional regularization loss is included to reduce motion jitter during silence. This loss can penalize large motion between adjacent animation frames, for example, when the sound volume is low.Furthermore, one or more embodiments can also accommodate lip distance loss to select key points on the upper / lower lips of the character and then minimize the difference between predicted and ground-truth distance between upper / lower lip key points.

[0009] When such image or video data is generated for various operations, a goal (or in some examples a requirement) for a character's representation—such as a human, robot, animal, or other such entity—may be to behave as realistically as possible. Such realistic behavior may involve various movements or actions in different states and under different conditions. For example, a character, such as one corresponding to the head region illustrated in a set of 100 images, might... Fig. Figure 1A illustrates that the character's mouth, face, and / or head will be animated to convey that the character is speaking, represented by audio data that can be provided for playback or other presentation alongside this animation. To make this animation appear as realistic as possible, simply moving the character's mouth to appear to utter the words in the speech may not be sufficient, as an emotional state or set of circumstances could affect how the character physically moves other elements (e.g., eyes, tongue, etc.) of their face while speaking. Simply animating the mouth to correspond to the formation of the words is not enough to convey realism in various situations, as the user could utter these words with very different intentions, emotions, and / or styles.For example, as illustrated in the first image 102, the character could be animated to utter the words with a happy emotion, such as where the character has come home to their family and is happy to see them. In a second image 104, the character might be angry that they have been summoned to a particular place and can express anger when conveying that they have arrived. In a third image 106, the character might feel a sense of disgust, such as where the character finds themselves in a place they really don't want to be but are forced to visit. In a fourth image 108, the character might feel a sense of sadness, as they may have announced their arrival but felt compelled to arrive due to undesirable circumstances.There can be any number of other emotions the character can experience, which are to be conveyed in various instances. To make this animation appear realistic, an animation system (or another image data generation or synthesis system, image data generation or synthesis component, image data generation or synthesis module, or image data generation or synthesis device) can assume or infer an emotional state and attempt to produce an animation of that character that not only corresponds to any audio spoken by that character but also conveys that utterance with an emotional behavior.

[0010] As illustrated, the communication of emotional behavior can involve a number of different but related movements. For example, an outer surface of the user's head—corresponding to skin 152 on the head, as in Fig. 1B illustrates this – being deformed to convey both speech and emotion. This can involve moving the character's lips 154 to match the formation of the uttered words, causing an animation of that character to utter the speech. As mentioned, the lip movement can also be determined based, at least in part, on that character's emotional state while speaking. There can be other aspects of the skin (or head) that may exhibit related behaviors, also based, at least in part, on the emotional state. For example, the amount or type of head movement (e.g., turning or tilting the head) may vary with different emotional states. Furthermore, aspects such as certain wrinkles 156 or lines in the character's skin may change for different emotions, perhaps becoming more or less noticeable or taking on specific shapes.Similar emotion-based behavior can also be shown through other skin regions or features, such as the eyebrows 158, cheeks, and so on.

[0011] In addition to this outer skin or surface, there may be other aspects or characteristics of this character that can alter behavior with different emotions, which may only be somewhat related to the behavior of the skin or surface, such as due to physical or kinematic limitations of the character. For example, the character's eyes may be modeled at least somewhat separately from the face. The position of the eyes may depend on the position of the character's head or skin, since the position of the eyes within the character's eye sockets is relatively fixed, but the movement or orientation of the eyes may be at least somewhat independent of the skin's behavior. For example, this character, when angry, might focus directly on a person they are speaking to, whereas this character might look away from another person when feeling sad or guilty.Similarly, the degree of saccadic movement or the frequency with which a character shifts their eye focus may vary for different emotions. Therefore, it may be desirable to derive eye orientation at least somewhat separately from skin, head, or surface behavior.

[0012] Conventional eye-tracking solutions may not offer adequate performance. In at least one embodiment, pupil tracking can be performed from the input 3D acquisition data using an algorithm, such as an optical Lucas-Kanade flow algorithm, which provides a differential approach to estimating the optical flow, assuming that the optical flow is substantially constant locally, and solves the basic optical flow within this local neighborhood. In cases where blinking or obstruction occurs, or where at least one eye is no longer visible in the acquired image data, at least some degree of interpolation can be performed based on one or more earlier (or subsequent, if available) image frames.Such eye-tracking approaches can also capture small saccadic eye movements, which can help make eye movement appear more natural in rendered facial animation. This approach can accurately model eye movement without requiring image data that represents a picture (or pictures) primarily focused on the actor's eyes while speech is being spoken.

[0013] Similarly, the character's tongue (164) can move at least somewhat independently of the head within physical or kinematic limitations. The amount or type of tongue movement can vary with emotion, as a sad character might exhibit very little tongue movement, while an angry or excited character might exhibit a great deal of tongue movement, which may also differ in direction or pattern. Furthermore, tongue movement (164) can also be related to a native language, as different sounds may not be present in every language. An appropriate number of trait points can be allocated to a tongue mesh, allowing for realistic movement and behavior through, for example, mesh deformation.This number of points can be reduced or compressed (to a number such as, without restriction, 10 points) by a process such as principal component analysis (PCA) in order to reduce the amount of processing and memory required for tongue mesh deformation.

[0014] Other aspects or features of a character may also be modeled separately to enhance realism. For example, the character's jaw (166) may be modeled separately from the character's head. While jaw movement can be approximated by skin movement and deformation, it has been observed that for at least some systems or implementations, such inference may not be accurate enough to avoid the need for post-processing or manual cleanup of the generated animation. To improve accuracy, the jaw movement (166) may be modeled separately, as the jaw can move in many different directions by varying amounts for a similar skin condition, such as where the character has closed their lips, but capturing this movement based on skin deformation alone can be challenging.Other aspects, traits, or components of a character may benefit from being modeled separately, which may depend at least in part on the character type, as an animal, robot, or stranger might be modeled to have a different skeletal structure or kinematic capability. Different types or instances of the same character may also exhibit different behaviors or emotions, such as people of different ages, genders, backgrounds, or other such aspects.

[0015] In many cases, a user might not display just a single emotion or might exhibit different levels of one or more emotions. For example, a character with an "angry" emotion type might behave very differently depending on whether they are mildly annoyed or furious. A character might also display multiple emotions at once, such as a character who is both happy that their child has been accepted to college and sad that the child is leaving, thus realistically displaying traits associated with a combination of both emotions. There may also be characters who exhibit different behaviors for the same emotion, at least under certain circumstances. For example, a character might act differently when speaking to a stranger than when speaking to a partner, parent, or child.A character might also act differently in a professional setting than in a personal one. In some cases, such as for a game or film, an animator might simply want a character to display a particular look, style, or behavior for a specific emotional state. Accordingly, for at least some of the approaches presented herein, it may be advantageous to allow a user (or application or operation, etc.) to specify more than one emotion or combination of emotions. In some operations, a user (or other source) may also be able to specify weights of these different emotions to provide more accurate combinations of emotions.A user can also specify different emotions, combinations, or emphases at different times or emotional "keyframes" in the animation, such as where a character becomes increasingly sad or calm during a discussion. A user can also specify a style with which a character conveys an emotion, which can also vary over time, such as at different keyframes in the animation.

[0016] Approaches according to various embodiments can utilize at least some of these and other such aspects or features to provide facial animation that delivers realistic behavior under various emotional states for a wide variety of character types and input audio types. This can include, for example, audio-driven, full three-dimensional (3D) facial animation with emotion control. With such an approach, realistic animation can be generated without requiring manual input or post-processing—although this is possible if desired. Automating such animation can significantly reduce the amount of time, expertise, and cost required for manual (or at least partially manual) character animation.Audio-driven facial animation can provide an efficient way to create facial animation compared to traditional approaches, as only audio data is required to drive the animation of a given character. Previous attempts at audio-driven animation might animate the lower face for lip-syncing, but were unable to generate appropriate movement or behavior for other facial areas or features—such as the upper face, teeth, tongue, eyes, and head—that may be necessary for accurate behavioral representation. With earlier approaches, it was often necessary to employ additional manual or post-processing efforts to correct inaccurate behavior in the generated animation.Previous attempts to incorporate emotions into speech animation typically focused on a single type of emotion for a duration of speech that, in many situations, failed to capture or accurately represent natural changes or shifts in behavior, often requiring additional manual or post-processing efforts. Furthermore, models used in earlier approaches often failed to generalize between different speech or audio inputs, such as non-speech inputs (e.g., shouting, yawning, sobbing, etc.). In an attempt to address these issues, earlier approaches incorporated additional training data for speech-specific applications or added application-specific loss functions, increasing the complexity and cost of facial animation.

[0017] Approaches according to various embodiments can provide automated, audio-driven animation, such as full 3D facial animation, with variable emotion control, which can generalize across different languages ​​and / or non-verbal audio inputs. In at least one embodiment, one or more aspects of an animation pipeline are configured to include a pre-trained audio encoder containing one or more transformer layers. This encoder can be trained on a variety of different languages ​​and / or multilingual audio snippets to compute audio features from an input audio segment over a time axis. Computed audio features from each transformer layer can be linearly combined using learnable weights.The combined audio features can then be routed along the pipeline to one or more audio decoders to estimate various animation coefficients, which can be used by one or more renderers to render a character with a facial movement associated with an audio input. In at least one embodiment, the pre-trained audio encoder is frozen during training, which can reduce the likelihood of overfitting, and therefore only the decoder and the linear combination layer weights can be modified during training. Systems and methods can be used both offline and in real time and / or near real time (e.g., without significant delay).

[0018] In at least one embodiment, a collection of speech performances from one or more actors can be captured, uttering speech (e.g., specific sentences) in different national languages, different non-speech audio, emotions, emotional levels, combinations of emotions, or presentation styles, among other such options. Emotions, as an example of an input component vector supported by such a system, can include any suitable emotion (or similar behavior or state) capable of being represented, at least partially, by character animation, image synthesis, or rendering, which may include, among others, joy, anger, astonishment, sadness, pain, or fear. A data acquisition process can involve capturing, for example, 4D data, including multi-view 3D data, over at least one period of speech utterance.The reconstruction of this captured facial behavior can be performed not only for facial skin (or such a surface) but also for other articulable or controllable components, elements, or features, such as teeth, eyeballs, head, and tongue (and / or body features or components, such as limbs, fingers, toes, torso, etc.). The reconstruction can provide geometric deformation data in the time domain for each separately (or at least somewhat separately) modeled facial component (or other body component) or region. Such a reconstruction can provide a complete dataset for use in training, for example, a deep neural network (as in ). Fig. 2 illustrated), to perform a task such as 3D face animation.

[0019] In at least one embodiment, a frame-to-frame mapping can be used to generate a single animation frame output. For example, single-frame audio (e.g., approximately 0.52 s) can be processed with a corresponding single animation frame output. In the exemplary System 200 of Fig. 2. An audio input 202, which may include one or more raw audio, audio frames, a segment of audio within a current audio window, etc., can be provided as input to the deep neural network. The deep neural network can use an analysis network section 208 to analyze the audio and encode features that represent features of the audio in the audio data 202, such as those corresponding to a segment of speech. This analysis network section 208 can include one or more audio decoders and encoders for encoding audio features into a feature vector, which, as specified herein, can be one or more combined audio features from transformer layers. The feature vector can be provided as input to an articulation network section 210 of the deep neural network 206. In this example, a component vector 204 can be provided as input.In some implementations, multiple vectors can be provided, such as an "emotion" vector and a "style" vector, among various other options and combinations. Additionally, a single vector can be provided as input that includes both style and emotion data, for example, through a fusion process. An emotion vector can contain data for one or more emotions applicable to speech used for training, such as an emotion the voice actor has been instructed to use when speaking the speech captured in the audio data. In some cases, this might include data for a single emotion label, such as "anger," or it might include data for multiple emotions, such as "anger" and "sadness," as well as potentially relative weights of these two emotions.These labels and / or weights may have been initially provided to the voice actor, may have been determined after the speech was spoken, and / or may include updated labels after the speech spoken for an audio capture for a particular emotion has been heard, among other such options.

[0020] In at least one embodiment, the component vector 204 can correspond to a style vector, which is provided as input to the deep neural network 206 during training (and similarly during deployment). A style vector can contain data relating to any aspect of the animation or facial component movement that modifies how one or more points for one or more facial components should move for a given emotion or emotion vector. A style vector can also be viewed as a finer control of the emotion, where an emotion vector provides the tag(s) of the emotion(s) to be used and the style provides finer control over how the emotion(s) are expressed by the animation. Other approaches to determining style data can also be used.In different implementations, a single set of emotion and style vectors can be provided for a given audio clip, a set of vectors can be provided for each animation frame to be generated, or a set of vectors can be provided for specific points or animation frames (e.g., emotional key frames) where at least one emotion or style value or setting is to be modified relative to a previous frame.

[0021] In this system 200, the component vector 204 is fed into an articulation network section 210 of the deep neural network 206 at multiple levels, including at least one beginning and one end of the network to aid in the conditioning of the network. The deep neural network 206 can use a shared audio encoder and one or more (e.g., multiple) decoders for each facial component (e.g., facial skin, jaw, tongue, eyeballs, and head).During training, an output network section 212 of the deep neural network 206 can generate a set of head / jaw displacements, eyeball rotations, and skin / tongue vertex positions / or PCA coefficients of vertices 214 and / or motion vectors (or other movements or deformations) for individual feature points of the facial components, whether for every such feature point or only for those that have changed relative to a previous frame, among other such options. During training, these vertex positions / or PCA coefficients can be compared with ground-truth data, such as the original reconstructed facial data from the (e.g., 4D) image acquisition, to calculate an overall loss score. In at least one embodiment, a loss, such as an L2 loss, can be used for both the position and velocity of feature points in an output data representation.A loss function used to determine the loss value may include terms for position, motion, and opponent loss in at least one embodiment. This loss value can be used during backpropagation to update network parameters (e.g., weights and distortions) for the deep neural network 206. As noted herein, in at least one embodiment, the weights for the audio encoder can be frozen, so that backpropagation is used to update only one or more decoders. Once it is determined that the network converges to an acceptable or desired level of accuracy or precision and / or another training end criterion is met (e.g., processing all training data or completing a target or maximum number of training iterations), the trained network 206 can be made available or deployed for inference.

[0022] During inference, the network can receive the audio input 202 (e.g., in some embodiments, only audio data) as input and can derive a set of vertex positions 214 for various facial components (e.g., head, face, eyeballs, jaw, tongue), which can then be fed to a renderer 216 (e.g., a rendering engine of an animation or video synthesis system) to generate an animation frame 218, which can be one of a series of frames that provide the animation during presentation or playback.As discussed in more detail elsewhere herein, emotion or style vector data can also be provided as input to the deep neural network 206 if the generated vertex positions are to be modified in any way with respect to how the deep neural network 206 would otherwise derive the vertex positions based on the audio data, such as to convey a specific style or facial behavior to be used in deriving the vertex positions 214.

[0023] A deep neural network 206, as described in more detail later herein, can output a vector encoding positional or motional data for different points on a mesh for one or more facial components, and can feed this output vector (or other output, such as a global transformation matrix) to a renderer 216, which can apply these values ​​to one or more meshes for that character to perform the animation. This output matrix or vector can have a dimension corresponding to the features of the facial components, such as, for example, and without limitation, 272 facial feature points for the skin, five for the head, five for the jaw, two for the eyeballs, ten for the tongue (for example, using PCA compression), and so on.Such an approach can provide sufficiently smooth animation, so that additional smoothing or post-processing is unnecessary in at least many situations. However, a system may allow for the application of additional smoothing, for example, by enabling a user to specify one or more smoothing parameters.

[0024] The size of the audio data and / or an audio data window can be any suitable or appropriate size and may depend partly on the implementation, but it can include at least a period corresponding to one animation frame for a target frame rate (e.g., 60 Hz) and can include larger windows to accommodate portions of audio for nearby frames (e.g., before or after) to provide more accurate and smoother animation, as well as more precise emotion and / or style determination from the input audio. An exemplary system might use one-hot vector coding to represent different emotions or emotional labels, with the resulting emotion vectors potentially chained together in one or more layers.

[0025] In at least one embodiment, the output of such a mesh can include data specifying motion for various facial parts or components. The output can include data for components of a character beyond just facial features, such as arms, legs, torso, and the like. Such a system can also generate accurate motion data for a character for frames or scenes where the character's face is not visible or is only partially depicted. For facial parts with non-rigid deformation, such as skin and tongue, captured (e.g., 4D) motion data can be compressed, for example, using principal component analysis (PCA). This can allow a facial mesh with a large number of points, such as 60,000 points, to be represented by a vector with a much smaller dimension, which can correspond to 272 (or another number of) feature values.In such embodiments, the PCA weighting vectors can be used as a training representation. In one embodiment, a fully connected layer of emotion and / or style data can be merged into a smaller vector that can be inserted as a concatenation into individual layers of the mesh.

[0026] For facial features with rigid transformations, such as the head and teeth, a number (e.g., without restriction, 5, as in Fig. (Figure 1B illustrates) can be selected or identified from markers or feature points on the target mesh, and a positional delta of these points from the reference position can be used as a training representation from which a rigid transformation matrix can be computed. For a jaw, these five feature points might include points at each end of the jaw, a midpoint, and two intermediate reference points, although these intermediate points may not be necessary but can aid in fine-tuning motion control and noise reduction. For rotatable components, such as eyeballs, the system can use two rotation values ​​(e.g., pitch and yaw) to represent horizontal and vertical rotations relative to a standard orientation. At runtime or during inference, a full 3D facial animation output can be obtained from such a system.Such a system can also enable interactive control of the emotion or style of this speech animation by feeding various component vectors 204 into the network to allow modification or “fine-tuning” of the movement of each individual facial component and facial region, which can be part of a real-time or near-real-time process or as a post-process.

[0027] In at least one embodiment, a deep neural network 206 can have a per-frame convolutional neural network (CNN) architecture. A per-frame CNN architecture can receive as input audio windows of, for example, approximately 0.5 seconds, which may contain data for earlier and subsequent frames in a sequence, enabling the CNN to predict data for a middle frame in that sequence. Other architectures can also be used within the scope of the various embodiments, which can also provide a uniformity of animation based, at least in part, on a context provided to or determined by these architectures. While various architectures can provide adequate results, certain architectures may perform better under certain circumstances or for certain aspects of speech-driven facial animation.For example, a CNN-based architecture has worked well for real-time inference and can generate very reliable lip-sync movements.

[0028] In at least one embodiment, the deep neural network can be a transformer architecture. A transformer architecture can differentially weight a significant for each part of an input. The input data can be provided as a sequence but can be processed all at once, with an attention mechanism providing context for a position in the input sequence. Transformers can include an encoder architecture in which coding layers iteratively process an input. The encoder determines relevant features of the input and passes these codings to the next layer. Attention mechanisms can be included to weight the relevance of each part of the input and / or output. For example, the attention mechanism can accept input codings from the previous encoder and weight their relevance to each other to generate output codings.The encoder can then individually process each output encoding in subsequent layers. These output encodings are then passed to the next encoder as its input.

[0029] Fig. Figure 3 illustrates an exemplary system 300 representing one or more components that form the analysis network 208, which can be provided in at least one embodiment. In this example, an audio input 202 is received by the deep neural network 206. In at least one embodiment, the audio input 202 is raw audio input; however, different embodiments may include audio inputs that have undergone one or more pre- or post-processing operations. The audio input 202 is provided in this example to a CNN 302, which can form a section of the analysis network 208. The CNN 302 can be part of a multilayer convolutional feature encoder or at least provide information to it, which receives the audio input 202 and outputs one or more latent representations 304.The latent representations 304 can be provided for one or more time steps and fed to a transformer 306 to determine representations for the sequence of information associated with the latent representations 304 and to generate a set of context representations 310. In at least one embodiment, each layer 308 of the transformer 306 can generate a set of context representations 310 for the latent representations 304 provided by the CNN 302. For example, different embodiments can include approximately 25 different features, with the context representations for each layer 308 being developed for each of the 25 features.

[0030] In at least one embodiment, a combination layer 312 can include learnable weights that can be applied to linearly combine each of the output vectors for each layer 308 to generate an output 314. The weights of the combination layer 312 can be learned while the model is trained by backpropagation. This output 314 can then be used by the articulation network 210, which also includes one or more of the Fig. The component vectors shown in 204 can be received to generate position information for rendering a character model.

[0031] Various implementations mask the latent representations 304 before feeding them into the transformer 306. From there, contextualized representations can be created, for example, through a contrastive learning process in which different samples are contrasted against each other. The model can be pre-trained on unlabeled data and then fine-tuned with labeled data, among other options. Accordingly, systems and procedures can use one or more models to build contextual representations with self-awareness of capture dependencies.

[0032] In at least one embodiment, the CNN 302 includes temporal convolution, followed by layer normalization and an activation function. The audio input 202 can be a raw waveform that is normalized before processing. The transformer 306 can receive the output feature encodings. Each layer 308 can learn a number of features and output a vector for these assigned features. The total number of features per layer 308 can be approximately 10 features, approximately 15 features, approximately 20 features, approximately 25 features, approximately 30 features, or any reasonable number of features. Furthermore, the number of features can be within a range of approximately 10 to 20 features, approximately 15 to 25 features, approximately 20 to 30 features, or any other reasonable range.

[0033] In the combination layer 312, each vector can be weighted for each layer 308 and then linearly combined to produce the output 314. The weights can be learned during training and / or can be tuned or adjusted based on different applications. The weighted vector indicates features from which the transformer layer 308 is more useful and combines these features linearly according to this determination. For example, if a feature from layer 1 is f1 and a feature from layer 2 is f2, these are input into the combination layer 312, and the output is calculated as f = (f1·w1 + f2·w2) / (w1 + w2), where w1 and w2 are the learnable weights.

[0034] In at least one embodiment, the audio input 202 can be a short audio sequence, such as one that is less than one second long. For example, the audio input 202 can be approximately 0.5 seconds long. Such small audio samples can provide the advantage of enabling real-time or near-real-time applications. Furthermore, a given frame or audio clip can be independent of or otherwise reliant on a previous frame or audio clip. As a result, output animations can be "skipped" or followed along different sections of the audio clip without losing information that would otherwise be needed if previous frames were left.

[0035] Various embodiments of the present disclosure may also implement one or more techniques to improve output quality. For example, an additional regularization loss may be added to reduce motion jitter during silence. This loss will penalize large movement between adjacent animation frames when the sound volume is low. Accordingly, there may be a reduced probability of movement between frames that would be expected if there were no sound, since the character would likely not move their mouth, for example, when not speaking. Likewise, systems and methods may also apply lip-space loss to improve output quality. For example, key points on lips (e.g.,(an upper lip and a lower lip) and then a minimal difference between the predicted and ground-truth distance between the key points is selected. The loss can have higher weights when the ground-truth lip distance is small (e.g., when the mouth is closed). As a result, the output character model can provide improved realism by guiding or otherwise encouraging mouth closing at appropriate times.

[0036] One or more embodiments of the present disclosure can train the transformer 306 such that parameters of the encoder are locked during training, but parameters of an associated decoder (e.g., a convolutional decoder) are allowed to change as a result of the training. For example, a pre-trained model trained on diverse audio data, such as several different languages ​​or sounds, can be used, and the associated parameters of the encoder cannot change or otherwise be updated during the training process. In this way, overfitting can be reduced, and a smaller amount of training data can be used.

[0037] Embodiments of the present disclosure include the illustrated transformer 306 in the deep neural network 206, such as within the analysis network 208, for processing one or more audio inputs 202, which may correspond to a window or clip of an audio segment. As noted herein, the audio segment may be a raw audio segment or a processed audio segment. Additionally, in at least one embodiment, the audio input may be buffered or otherwise truncated before processing within the deep neural network 206 and / or as part of the deep neural network 206 in one or more preprocessing stages. For example, the deep neural network 206 may take an audio buffer of a given length, such as length 8320 (e.g., approximately 0.52 s of audio), to compute audio features along a time axis.In certain embodiments, a number of audio features are determined by one or more properties of the CNN 302 and / or the transformer 306. For example, existing systems can evaluate approximately 50 different features. Various embodiments of the present disclosure can evaluate more or fewer features, such as approximately 25 features. These computed features can then be evaluated in each of the layers 308 of the transformer 306. Subsequently, the computed audio features from each transformer layer 308 can be linearly combined using learnable weights (e.g., in the combination layer 312), the weights being learned during one or more training operations. Additionally, in various embodiments, the weights can be tailored for specific applications based on one or more desired outputs.The linearly combined audio features are then passed to the audio decoder for estimating animation coefficients. As noted, in at least one embodiment, while parameters of the audio decoder can be adjusted or modified based on training information, systems and methods can use a pre-trained audio encoder that is locked or otherwise unmodifiable during training. That is, the pre-trained audio encoder is frozen, and only the decoder weights are trained. In at least one embodiment, the decoder can be a convolutional decoder. Accordingly, due to the non-autoregressive nature, systems and methods can be used for both online and streaming applications (e.g., real-time and / or near-real-time).

[0038] The systems and methods of this disclosure can be incorporated into one or more user interfaces that can be used to tune or otherwise modify a facial or body animation for one or more computer-animated characters. For example, a user interface can be provided that displays an animated representation of the character at different times and provides one or more inputs for the user to modify or adjust various parameters, such as tuning or otherwise changing various parameters of the one or more component vectors described herein. In various embodiments, the interface can be used for both training and inference purposes.For example, the user can evaluate reconstructions on the interface along with values ​​assigned to one or more features of the component vectors to label data that can be used for training purposes. In other embodiments, a user can also use such an interface to modify the displayed reconstruction.

[0039] The interface may include the ability for the user to evaluate reconstructions and animations at a given time over the duration of an audio clip. As noted herein, since various implementations can generate animations over small time clips (e.g., about 0.5 seconds) and also do not use prior information when generating animations, the user can "skip" or otherwise navigate through different sections of the audio clip without losing the reconstruction quality due to the lack of information from one or more preceding frames. For example, a user can begin viewing an animation at a time corresponding to 20 seconds into the clip and then jump to a time corresponding to 2 minutes into the clip without losing the reconstruction quality due to not evaluating the frames between 20 seconds and 2 minutes of the clip.Since different time points within the audio clip can be assigned without different outputs due to the context vector, various implementations provide improved review and viewing by allowing skipping or otherwise following through the audio clip.

[0040] As mentioned, such an interface can be used at inference time as a kind of post-process, which can also be used for continuous learning in at least some implementations. For example, a user can view animation playback generated by the user through this interface, where the character's animation is presented. If the user thinks the animation contains too much intensity for the situation, the user can adjust one or more style selectors to reduce the intensity or modify one or more other parameters and have the animation frame(s) re-rendered. If the user detects a hint of sadness in the character's speech that is not captured in the animation, the user can adjust this setting as well.In some implementations, a user may also be able to provide a style input for customizing specific feature points or facial components in the display. For example, the user can use a pointer to grab and move the character's lip, and this information can be used as a style input to re-render the animation. Other modifications can also be provided, such as head movement, head tilt, eye movement or focus, or other such changes that can be conveyed through emotions or style input to re-render (or update or synthesize) the animation. Various other animation control parameters can also be specified through such an interface, which can affect the final rendering.

[0041] Various systems can also support retargeting. Retargeting allows the movement of one character to be mapped to the movement of another character, enabling the creation of similar animation for similar emotions and / or styles. For example, retargeting can be applied to one or more (e.g., all) facial components, such as skin, jaw, tongue, eyeballs, etc. These facial components can be retargeted to more closely match or resemble the same facial components of a target or custom character. Furthermore, an interface can be advantageous in a remapping context, where different characters might express emotions or styles in slightly different ways.A user may be able to load different characters into this interface and view how a retargeted rendering would appear for that character, and may then modify one or more aspects or a style of movement or behavior for that specific character or type of character.

[0042] Fig. Figure 4 illustrates an exemplary training process 400 that can be used to train a neural network for a task such as facial animation, according to at least one embodiment. It is understood that for this and other processes presented herein, additional, fewer, or alternative operations may be performed in similar or alternative sequences, or at least partially in parallel, within the scope of the various embodiments, unless specifically stated otherwise. Although this example relates to facial animation, it is further understood that various other such tasks may benefit from aspects of such a training process within the scope of different embodiments. In this process, audio and 4D image data (and / or other image data, such as 2D image data, 3D image data, etc.) are used.) for one or more actors uttering speech, including the same or different words or content, using different emotions and / or styles, 402 or otherwise. In at least some embodiments, the actor uttering the speech is instructed to utter the speech using at least one specified emotion or style or other parameter that can be modeled and represented using one or more component vectors, and is further to “negotiate” the parameter during utterance in order to attempt to capture image or video data that represents realistic physical movement or behavior of one or more facial or body components during utterance with the specified parameter (e.g., emotion, style, etc.).This recording can be performed for any different number of emotions or phrases and can include sufficient instances of the utterance of each emotion to allow for accurate training of, for example, a deep neural network.

[0043] A reconstruction of facial animation can be performed to provide a type of ground-truth data, whereby a character mesh or other representation is distorted based on the captured image data to provide a reference to how facial components actually moved or distorted during the utterance of a specific speech with a specific emotion and / or style. Furthermore, the ground-truth information can correspond to the movement of various facial features, such as lips, jaw, etc., to determine whether an accurate movement corresponds to the utterances associated with the audio data.

[0044] In at least one embodiment, a pre-trained transformer model can be provided. The pre-trained transformer model can be pre-trained on audio data that may contain a variety of different languages ​​and can be used as an audio encoder within one or more pipelines. The pre-trained transformer model can generate a set of encodings that are used as input to different layers of the transformer model and / or to a decoder associated with the pre-trained transformer model. In at least one embodiment, providing the pre-trained transformer model involves locking or otherwise preserving initial weights or parameters of the pre-trained transformer model so that these weights or parameters are not changed or updated during a training process.

[0045] At least some of the training data can then be made available for use in training a deep neural network. This can include, for example, a window of audio and component data, such as emotion data, for each animation frame to be generated. As mentioned, the audio and emotion data (as well as potentially any state data) can be used to generate feature vectors that can be provided as input to the network during training. As mentioned, sequence-to-sequence mapping can be used to obtain a sufficiently long temporal context for this input data, which can be advantageous when generating physically or behaviorally accurate animation.

[0046] The neural network can generate an output vector 412 when receiving and processing this data, which corresponds to linearly combined output vectors for each frame of the pretrained transformer model. For example, the input training data can be processed by one or more networks to extract one or more features, with these features being provided as input to the pretrained transformer model. The pretrained transformer model can then output vectors for each layer of the model and pass these vectors to a combination layer to linearly combine the vectors from each layer into the output vector.

[0047] Once generated, the output vector can be provided to a decoder 414, which may be part of the pretrained model or a separate decoder, to update weights of the decoder model based on the output vector. The decoder can then produce an output for use by one or more networks 416 to generate motion vectors for various points associated with the one or more features of the output vector. For example, the neural network can generate a set of motion vectors (or vertices or deformation values, etc.) for one or more facial feature points of one or more of the character's facial feature points. This may involve generating motion vectors for every facial feature point, or only for those feature points that experience at least a certain amount of motion, among other such options.The network can then be used for inference or otherwise provided, 418 such as to generate facial animation data from input audio (and / or from one or more emotion vectors and / or one or more style vectors).

[0048] Fig. Figure 5 illustrates an exemplary process 500 for generating facial animation data using such a trained deep neural network, according to at least one embodiment. In this example, audio data 502 is received, which includes at least some speech data for which a character is to be animated. In addition to this audio data, there may be one or more component vectors containing adjustments or settings—such as emotion or style of animation—to be applied to the speech animation, which can be received as additional input 504. This audio data and any received adjustment data can be provided to a trained neural network 506, such as one trained using a process that is similar to the one described in terms of Fig. This process is similar to that described in section 4. This process can then receive motion vectors (or vertices, etc.) derived from this neural network for individual animation frames, 508 where these motion vectors specify movement for feature points corresponding to a variety of facial components (or other body components), such as a head, jaw, skin, eyeballs, or tongue, among other such options. These motion vectors are derived to provide a realistic animation for a character uttering the speech represented in the appropriate section of the audio data, and to allow separate modeling and behavior of these separate components within kinematic, structural, and / or other such constraints.In this example, these motion vectors can then be provided to a renderer (or other such system, service, device, or process) to render one or more frames of facial animation 510 by, for example, deforming one or more meshes for these facial components according to these motion vectors. This animation can then be made available for purposes such as presentation or storage 512, among other such options.

[0049] Fig. Figure 5B illustrates an exemplary process 520 for generating facial animation data using such a trained deep neural network, according to at least one embodiment. In this example, audio data is received that includes an utterance by a speaker 522. The audio data may include spoken phrases in a variety of languages ​​and / or non-verbal spoken emissions, such as shouting, crying, sobbing, heavy breathing, and / or the like. In at least one embodiment, the spoken phrases may be accompanied by a particular style, which may include singing or the like. Consequently, spoken phrases may refer to verbal emissions and not necessarily only to speech.

[0050] One or more neural networks can input a transformer-based audio encoder to compute a weighted vector that specifies a variety of features associated with the audio data. The variety of features can be determined by one or more neural networks, such as a CNN, that extract feature information from the audio input before providing that information to the transformer-based audio encoder. For example, the CNN can learn which features of the input audio data are relevant for a given task, identify those features, extract them, and then provide them to the transformer-based audio encoder.

[0051] In at least one embodiment, one or more animation vectors can be computed using the weighted vector and one or more component vectors 526. The component vectors can correspond, among other options, to an emotion or style vector that modifies various features of an output animation. For example, an emotion corresponding to "angry" can cause an output animation to make a character tense their jaw and / or furrow their brow. Similarly, a style corresponding to "singing" can cause the animation to change its mouth position to project different words or to keep its mouth open longer to emphasize certain notes or the like. Based on the one or more animation vectors, a digital character representation can be rendered 528.

[0052] The systems and methods of this disclosure can enable a deep neural network to effectively process human speech and generalize across different speakers. Such a process can also allow a network to detect variations in training data that cannot be explained by the audio alone, which may relate to an obvious emotional state, style, or other component driving animation. A three-way loss function, such as the one presented herein, can also help ensure that the network remains temporally stable and responsive under animation, even with highly ambiguous training data. Such a process can generate expressive 3D facial movement from audio in real time or near real time (e.g., without significant delay) and with low latency.To maintain independence from the details of the downstream animation system, such a system can output the per-frame positions of the control vertices of a fixed-topology face mesh. Alternative encodings, such as hybrid or nonlinear rigs, can be introduced in later pipeline stages if required for compression, rendering, or editability. A sample mesh can be trained using three to five minutes of high-quality footage obtained through conventional view-based performance capture methods. Such a process has been observed to successfully model the speech style not only of a single actor but also of other speakers of different genders, accents, or languages.This flexibility can be useful for various applications or operations, such as in-game dialogue, cost-effective localization, virtual reality, augmented reality, mixed reality, and telepresence, among other options. Such an approach can also prove useful for incorporating small script changes, even within the kinetics.

[0053] An exemplary and non-restrictive deep neural network, incorporating at least one CNN, consists of one special layer, ten convolutional layers, two fully connected layers, and at least one transformer model with approximately six layers, which may be subdivided into three conceptual parts, as shown in Fig. 2 and Fig. Figure 3 illustrates this. In at least one embodiment, the systems and methods include a pipeline that receives input audio, processes the input audio through one or more CNN layers, processes an output from the CNN layers through one or more transformer layers, processes an output from the transformer layers through a linear combination layer, and then provides an output to an articulation layer. Different outputs from different layers can be routed to different other layers, as noted herein, such as providing each output from each transformer layer to the combination layer. As a result, different embodiments can have a variety of different output vectors from different layers that are processed, combined, modified, and / or the like.Data for an input audio window can be fed into a formant analysis network to generate a time-variant sequence of speech features that subsequently drive articulation. Through training, the convolutional layers learn to extract short-term features relevant for facial animation, such as innation, stress, and specific phonemes. Their abstract, time-variant representation can be the output of the final convolutional layer.

[0054] The result can be fed to a transformer containing an encoder. The encoder can be a pre-trained audio encoder trained on a variety of different audio data, such as multiple languages, mixed languages, and the like. Parameters of the audio encoder can be frozen, while other system parameters are modified during training. In this example, an articulation network can also form part of the deep neural network, which consists of five further convolutional layers that analyze the temporal evolution of features and ultimately decide on a single abstract feature vector describing the facial posture in the center of the audio window.As a secondary input, the articulation network accepts a (learned) description of emotional state and / or style (which can be referred to as a component vector) to distinguish between different facial expressions and speech styles. The emotional state, alone or with style data, can be represented as an E-dimensional vector that is directly chained to the output of each layer in the articulation network, allowing subsequent layers to modify their behavior accordingly.

[0055] The network architecture can, in at least one embodiment, include the CNN of the analysis network 208 (e.g., CNN 302), which has a total of seven 1D convolutions (along the time axis). The convolution kernel size of the seven layers can be 10, 3, 3, 3, 3, 2, 2 each. The convolution step for the seven layers can be 5, 2, 2, 2, 2, 2, 2. The output of the convolution can have a shape of 25 (time dimension) x 512 (feature dimension), which is fed to a feature projection layer to change its shape to 25 x 768. This 25 x 768 shaped output can be fed to the transformer layers (e.g., layers 308) (which may contain, for example, six layers, twelve layers, or any other reasonable number of layers), which also output a feature with a 25 x 768 shape from each layer. These outputs can then be passed to a linear combination layer (e.g., layer 312).

[0056] An exemplary articulation mesh can output a set of 256 + E + S abstract features that together represent the desired facial posture—e.g., E for the dimension of the emotion vector and S for the dimension of the style vector. These outputs can be fed to an output mesh to generate the final 3D positions of a set of control vertices in a tracking mesh in at least one embodiment. Furthermore, in various embodiments, the output mesh computes PCA coefficients for the facial mesh (e.g., a total of 140), the tongue mesh (e.g., a total of 10), and rigid transformation coefficients for the eyeballs (e.g., a total of four) and / or the jaw (e.g., a total of 15). The output mesh can be implemented as a pair of fully connected layers that perform a simple linear transformation on the data.The first layer maps the set of input features to the weights of a linear basis, and the set of second layers computes the final PCA coefficients for face and tongue, rotation values ​​for eyeballs, and translational displacements for jaw and head. If the output mesh computes 3D vertex positions as output, the second layer can be initialized, for example, to 150 pre-computed PCA components that together explain approximately 99.9% of the variance seen in the training data. In one or more embodiments, the two linear transformation layers of the output mesh can have an architecture such that the first transformation layer transforms features of size 256 + E + S into a feature of the form 169. Then, the second transformation layer takes an input of the form 169 and outputs coefficients of the form 169 (140 + 10 + 4 + 15). The transformation layer weights can be initialized randomly.

[0057] A primary input to such a network is a speech audio signal, which can be converted into a format such as 16 kHz mono audio before being fed into the network. The volume of each voice track can be normalized to utilize a full [-1, +1] dynamic range, but such a system may or may not employ other types of processing, such as dynamic range compression, noise reduction, or pre-emphasis filters.

[0058] In one example implementation, audio with a duration of 520 ms was used as input, representing, for example, 260 ms of past and future samples relative to the desired output position. This value was chosen to capture relevant effects, such as phoneme coarticulation, without providing too much data that could lead to over-matching.

[0059] Deriving facial animation from speech can be an inherently ambiguous problem, as the same sound can be produced with very different facial expressions. This is especially true for the eyes and eyebrows, as they have no direct causal relationship to sound production. Such ambiguities are also problematic for deep neural networks, since the training data inevitably contains cases where nearly identical audio inputs are expected to produce very different output attitudes. If a network has nothing to work with besides the audio, it will learn to output the statistical mean of the conflicting outputs.

[0060] One exemplary approach to resolving such ambiguities is to introduce at least one secondary input into the network. A small amount of additional, latent data can be assigned to each training sample, providing the network with sufficient information to unambiguously derive the correct output attitude. This additional data can encode all relevant aspects of the animation in the vicinity of a given training sample that cannot be derived from the audio itself, including different facial expressions and coarticulation patterns. This secondary input can include a predefined label and can represent at least one emotional state of the actor.Besides resolving ambiguities in the training data, such a secondary input can also be very useful for derivation, as it allows a system to mix and match different emotional states with a given voice track to provide powerful control over the resulting animation.

[0061] In addition to or as an alternative to relying on predefined labels, a system according to at least one embodiment can employ a data-driven approach in which the network automatically learns a short representation of the style as part of the training process. This enables the system to extract meaningful emotional states even from material contained within the character, provided a sufficient range of emotions is present. In at least one embodiment, a style state can be represented by an S-dimensional vector, where S is a tunable parameter that can be set to a value, such as, without restriction, 16 or 24, and the components are initialized to random values ​​drawn from a Gaussian distribution.Such a vector can be assigned to each training sample, with the matrix that stores these latent variables referred to here as a style database. The style data can be appended to the list of activations of all layers of the articulation network, making it part of the computation graph of the loss function, and as a trainable parameter, it can be updated along with the network weights during backpropagation. The dimensionality of S is a trade-off between two effects in this example. If S is too low, the styles cannot distinguish variations in the training data, resulting in a weak audio response. If S is too high, styles can become too specialized to be useful for general inference.

[0062] Information provided by the audio may be limited to short-term effects within, for example, the 520 ms interval specified in the design. Consequently, one technical approach to prevent styles from containing duplicate information is to prohibit them from including short-term variation. Having styles focus on longer-term effects may also be desirable for inference, as it may be desirable for the network to generate reasonable animation even if the emotional state remains constant. This requirement can be addressed by introducing a dedicated normalization term into the loss function to penalize rapid variations in the style database, which can lead to incremental smoothing of emotional states during training.One possible limitation of such an approach is that aspects such as blinking and eye movement may not be modeled correctly, as they do not correlate with the audio and cannot be represented using the slowly varying emotional state.

[0063] In some implementations, emotional and stylistic states can be attached to all layers of the articulation network to significantly improve practical results. This is because emotional and stylistic states can control animation at multiple levels of abstraction, and the higher levels can be more difficult to learn. Connecting to the earlier layers provides nuanced control over subtle animation features such as coarticulation, while connecting to the later layers provides more direct control over output postures. Early stages of training can focus on the latter, while later stages can focus on the former once the individual postures are reasonably well represented.

[0064] In one approach to training a deep neural network, an unstructured network can be reconstructed using texture and optical flow data from, for example, the nine images acquired for each frame. A fixed-topology template mesh, created prior to acquisition using a separate photogrammetry pipeline, can be projected onto the unstructured network and mapped to the optical flow. The template mesh can be tracked across the process, and all issues are semi-automatically resolved, similar to how a tracking artist would handle this in software. The head position and orientation can be stabilized using key vertices from the tracked mesh. Finally, the vertex positions of the mesh for each frame in the acquisition can be exported.These positions - or more precisely the deltas from a neutral stance - can be target outputs of this network if an audio window is provided during training.

[0065] For each actor, a training set can consist of at least two parts: pangrams and material contained within the character. In general, the quality of inference can increase as the training set grows, but a small training set can be highly desirable due to the cost of acquiring high-quality training data. In at least one embodiment, it has been empirically determined that approximately three to five minutes per actor represents a practical sweet spot. A pangram set may attempt to cover the set of possible facial movements during normal speech for a given target language, such as English. The actor speaks one to three pangrams, for example, sentences designed to include as many different phonemes as possible, in several different emotional tones to provide good coverage of the expressive range.A character-based material set can leverage the fact that an actor's performance of a character is often highly distorted in terms of emotional and expressive range for various dramatic and narrative reasons. In the case of a film or game production, this material may be compiled from the preliminary version of the script. Only the shots deemed to support the various aspects of the character are selected to ensure that the trained network produces an output that remains in character, even if the inference is not perfect or if a completely novel or uncharacteristic voice is introduced.

[0066] Given the potentially ambiguous nature of the training data, an effort can be made to define a meaningful loss function to be optimized. In at least one embodiment, a specialized loss function can be used, consisting of three distinct terms: a position term to ensure that the overall position of each output vertex is approximately correct, a motion term to ensure that the vertices exhibit the correct type of motion under animation, and a regularization term to prevent the style database from containing short-term variation.

[0067] Optimizing multiple loss terms simultaneously can be challenging in practice, as the terms may vary significantly in size and their equilibrium can shift unpredictably during training. One solution is to assign a predefined weight to each term to ensure none are neglected during optimization. However, selecting optimal weight values ​​can be a tedious trial-and-error process that must be repeated whenever the training set changes. To address these issues, a normalization scheme can be employed that automatically balances the loss terms according to their relative importance. As a result, an equal effort can be applied to optimizing each term, eliminating the need to specify additional weights.

[0068] One error metric that can be used is the mean of the squared differences between the desired output y and the output produced by the network ŷ. For a given training sample x, this can be expressed using the position term P(x): P(x)=1N∑i=1N(y(i)(x)−y^(i)(x))2

[0069] Here, N represents the total number of output features, including the 3D position of skin / tongue mesh vertices, rotation values ​​of eyeballs, and translational shift for jaw / head, and y (i) denotes the i-th scalar component of y = (y (1) , y (2) , ..., y (N) ). For example, N would be 61019 (e.g., 60000 for facial mesh, 1000 for tongue mesh, 4 for eye rotation, and 15 for jaw translation).

[0070] Even though the position term ensures that the network's output is approximately correct at a given time, it may not be sufficient to produce high-quality animation in all cases. Training the network with the position term alone has been observed to lead to a considerable degree of temporal instability, and the response to individual phonemes is generally weak. Accordingly, a network can also be optimized with respect to vertex movement: a given output vertex should only move if it also moves in the training data, and it should only move at the correct time. A system can thus address vertex movement as part of the loss function.

[0071] One approach to training neural networks is to repeat the training data in minibatches, with each minibatch consisting of B randomly selected training samples x1, x2, ..., x Bconsists of. To account for vertex motion, we plot the samples as B / 2 time pairs, each consisting of two adjacent frames. The operator m[·] can be defined as the finite difference between the paired frames, which allows us to define the motion term M(x) as: M(x)=2N∑i=1N(m[yi(x)]−m[y^(i)(x)])2

[0072] The factor 2 appears in this equation because M is evaluated once per time pair.

[0073] Additionally, it can be advantageous to ensure that the network correctly assigns short-term effects to the audio signal and long-term effects to the emotional state. One approach can define a regularization term for the emotion / style database using the same finite difference operator as above: R'(x)=2E∑i=1Em[e(i)(x)]2

[0074] Here, e( i)(x) is the i-th component stored by the emotion database for training sample x. It can be noted that this definition of the emotion / style database does not explicitly prohibit the inclusion of short-term variation—instead, it discourages excessive variation on average. This can be significant in at least some cases, as the training data may occasionally contain legitimate short-term changes in emotional state, and it may be undesirable for the network to incorrectly attempt to explain them based on the audio signal.

[0075] One caveat with Equation 3 is that R'(x) can be arbitrarily brought close to zero by simply decreasing the size of e(x) while increasing the corresponding weights in the net. Based on the idea of ​​batch normalization, this trivial solution can be eliminated by normalizing R'(x) with respect to the observed size of e(x): R(x)=R'(x)(1EB∑i=1E∑j=1At(xj)2)

[0076] To balance these three loss terms, one approach is to leverage the properties of an Adam (or other) optimization algorithm used to train the network. Adam updates the network weights according to the gradient of the loss function, which is normalized component-wise based on a long-term estimate of its second raw moment. This normalization makes the training resistant to differences in the magnitude of the loss function, but this only holds true for the loss function as a whole—not for the individual terms. One approach is to perform a similar normalization for each loss term individually. Using the position term as an example, the second raw moment of P(x) can be estimated for each minibatch, and a moving average v can be calculated. P tmaintained across successive minibatches, as can be achieved by the following: vtp=β.vt−1p+(1−β).1B∑j=1BP(xj)2

[0077] Here, t denotes the minibatch index and β is a decay parameter for the moving average, which can be set to a value, such as 0.99 without restriction. The system can vtp=0 initialize and correct the estimate to account for start-up variance, in order to vtp=vtp / (1−βt) to obtain the average P(x). The average can then be calculated over the current minibatch and the value according to vtp: lP=(1B∑f=1BP(xj))(v^tP+ε)

[0078] In equation 6, ε is a small constant that takes a value such as 10 - 8This can be adjusted to avoid division by zero. By repeating equations 7 and 8 for M(·) and R(·), a final loss function can be obtained as a sum of the individual terms l = l. P + l M + l R can be expressed. In some embodiments, the importance of the loss terms can be further fine-tuned by additional weightings.

[0079] Similarly, losses during training can also be associated with lip-distance regularization and volume-based stability regularization, as described herein. For example, key points on lips (e.g., an upper lip and a lower lip) can be selected, and then a minimal difference between the predicted and ground-truth distances between the key points can be chosen. The loss can have higher weights when the ground-truth lip distance is small (e.g., when the mouth is closed). Furthermore, a loss can be added to penalize large movement between adjacent animation frames when the sound volume is low. Accordingly, the final loss can be further fine-tuned to include losses for one or both lip-distance and volume-based stability.

[0080] In at least one embodiment, a random time shift can be used for training samples to improve temporal stability and reduce overfitting. Whenever a minibatch is presented to the network, the input audio window can be randomly shifted by up to 16.6 ms in any direction (± 0.5 frames at 30 FPS). To compensate for this, the same shift can be applied to the desired output position by linear interpolation. Both training samples in a temporal pair can be shifted by the same amount, using different random shift amounts for different pairs. In some embodiments, cubic interpolation of outputs can be used instead of, or in addition to, linear interpolation.

[0081] To improve generalization and avoid overfitting, multiplicative noise can be applied to the input of individual convolutional layers. The noise can be the same size for each layer and can be applied on a per-feature-map basis, so that all activations of a given feature map are multiplied by the same factor. Identical noise can be applied to paired training samples to obtain a meaningful motion term. One formula for this noise is 1.4 N(0,1)No other type of noise or gain can be applied to the training samples besides input / output time shifting and multiplicative noise within the network. However, some approaches can perform operations such as displaying volume, adding reverberation (both long and short), performing time stretching and pitch shifting, applying nonlinear distortion, random equalization, and scrambling phase information, among other such options.

[0082] After training, a deep neural network can be evaluated at arbitrary time points by selecting the appropriate audio window, resulting in facial animation at the desired frame rate. The latency of such an approach can depend, at least in part, on the audio window size, which can extend a period into the past and / or future. Coarticulation can impose a lower bound on look-ahead; it has been observed that look-ahead can be limited to values ​​such as 100 ms during training with minimal degradation in quality, although some coarticulation effects may be longer. Shortening the look-ahead further than this can lead to a rapid decline in perceived responsiveness in certain cases, so a realistic lower bound for the latency of an implementation can be set at approximately 100 ms.

[0083] When facial expression is inferred for novel audio, the network can be provided with an emotional state vector and / or a style vector as secondary input, which can also be part of a single emotion vector. As part of the training, the network can learn a vector (e.g., a latent E-dimensional vector) for each training sample, and this emotion database can be used to obtain robust emotion vectors that can be used during inference.

[0084] During training, the network may attempt to extract latent information—e.g., anything not inferable from the audio alone—into an emotion / style database. However, this decomposition can introduce some degree of crosstalk between articulation and the overall expression. In practice, many of the learned emotion / style vectors may be applicable only within the vicinity of their corresponding training frames and are not necessarily useful for genetic inference. In at least one embodiment, a process can search for robust emotion / style vectors using a three-operation process. One problem encountered with many learned emotion vectors is that they emphasize mouth movement: if such a vector is used as a constant input when inference is performed on novel audio, the apparent mouth movement may be suppressed.One approach is to select some audio windows from a validation set that contain bilabial sounds and some that contain vowels for which the mouth should be closed or open, respectively. The emotion / style database can then be scanned for vectors that exhibit the desired behavior for all selected windows. Performing this preliminary culling for Character 1 resulted in 100 candidate emotion vectors for further consideration, and this response can vary with different emotion vectors.

[0085] A second operation in this exemplary culling process involves replaying the validation audio tracks and inspecting the facial movement inferred from each of the candidate emotion / style vectors. At this stage, vectors resulting from suppressed or incorrect, unnatural movement can be discarded, meaning the vector may have short-term effects. This stage narrowed the set to 86 candidate emotion vectors for Character 1. As a third and final operation in this example, inference can be performed on several seconds of audio from another speaker and vectors with a subdued or unnatural response. For Character 1, this operation left 33 emotion vectors.

[0086] The output of the network can be examined for several novel audio clips with each remaining emotion / style vector and a semantic meaning (e.g., "neutral," "amused," "surprised," etc.) assigned to each, at least partially dependent on factors such as the emotional state they convey. Which semantic emotions remain may depend on the training material, and it may not be possible to extract, for example, a "happy" emotion if the training data does not contain enough such material to generalize to novel audio. Even after removing all but the best-performing emotion vectors, there may still be considerable variation to choose from. Emotion vectors obtained in this way have been observed to perform well under interpolation, e.g.,Swiping from one emotion vector to another tends to produce natural-looking results. It may therefore be possible to vary the emotional state during inference based on high-level information from a machine learning system or through manual keyframing.

[0087] The resulting facial animation can be highly stable. Primary sources of this temporal stability may be the motion term l. Mand time-shift amplification, but even with these techniques, a small amount of jitter may still remain, such as in the lip region at a 4 ms timescale for some inputs. This can result from aliasing between adjacent audio frames around features like stops and plosives. This can be at least partially mitigated using at least one set of feedback: The network is evaluated twice for a given animation frame, with a time interval (e.g., 4 ms) between each evaluation, and the predictions are averaged.

[0088] As mentioned, such an approach can also support retargeting. During model training, the output mesh can be specialized for a specific mesh. For many operations, it may be desirable to drive several different meshes using audio input. The approaches discussed here can support retargeting of deformation or the transfer of deformation behavior between characters or for the same character at different life stages, among other such options.

[0089] As discussed, aspects of various approaches presented herein may be simple enough to be executed in real time on a device such as a client device, like a personal computer or a game console. Such processing may be performed on or for content that is generated on or received by that client device, or received from an external source, such as streaming data or other content received over at least one network. In some cases, the processing and / or determination of this content may be performed by one of these other devices, systems, or entities, which are then made available to the client device (or other such recipient) for presentation or other such use.

[0090] As an example, it illustrates Fig. 6 An exemplary network configuration 600 that can be used to provide, generate, modify, encode, process, and / or transmit image data or other such content. In at least one embodiment, a client device 602 can generate or receive data for a session using components of a control application 604 on the client device 602 and data stored locally on that client device. In at least one embodiment, a content application 624 running on a server 620 (e.g., a cloud server or edge server) can initiate a session associated with at least one client device 602, such as a session manager and user data stored in a user database 636, and can cause content, such as one or more digital assets (e.g.,Object representations) from an asset repository 634 are determined by a content manager 626. A content manager 626 can work with an audio-to-face module 628 to create or synthesize new objects, digital assets, or other such content to be made available for presentation via the client device 602. In at least one embodiment, this audio-to-face module 628 can use one or more neural networks or machine learning models that can be trained or updated using a training module 632 or system located on or communicating with the server 620.This may involve training and / or using a diffusion model 630 to generate content tiles that can be used by an audio-to-face module 628 to, for example, apply a non-repeating texture to a region of an environment for which image or video data is to be presented via a client device 602, which may involve using one or more renderers 630. At least some of the generated content can be transferred to the client device 602 using a suitable transfer manager 622 for delivery by downloading, streaming, or another such transfer channel. An encoder may be used to encode and / or compress at least some of this data before it is transferred to the client device 602.In at least one embodiment, the client device 602, which receives such content, can provide this content to a corresponding control application 604, which may also or alternatively include a graphical user interface 610, an audio-to-face manager 612, and a renderer 614 for use in providing, synthesizing, modifying, or using content for presentation (or other purposes) on or by the client device 602. A decoder can also be used to decode data received over the network(s) 640 for presentation via the client device 602, such as image or video content through a display 606 and audio, such as sounds and music, through at least one audio playback device 608, such as a loudspeaker or headphones.In at least one embodiment, at least part of this content may already be stored, rendered, or accessible on the client device 602, so that transmission over the network 640 is not required for at least this part of the content, for example, if this content may have been previously downloaded or stored locally on a hard disk or optical disk. In at least one embodiment, a transmission mechanism, such as data streaming, may be used to transfer this content from the server 620 or the user database 636 to the client device 602. In at least one embodiment, at least part of this content may be received, augmented, and / or streamed from another source, such as a third-party service 660 or another client device 650, which may also include a content application 662 for generating, extending, or providing content.In at least one embodiment, sections of this functionality can be performed using multiple computing devices or multiple processors within one or more computing devices, such as a combination of CPUs and GPUs.

[0091] In this example, these client devices can include any suitable computing devices, such as a desktop computer, a notebook computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, AR glasses, a portable computer, or a smart TV. Each client device can submit a request over at least one wired or wireless network, such as the internet, Ethernet, a local area network (LAN), or a cellular network, among other such options. In this example, these requests can be submitted to an address associated with a cloud provider that can operate or control one or more electronic resources in a cloud provider environment, such as a data center or server farm.In at least one embodiment, the request can be received or processed by at least one edge server located at a network edge and outside at least one security layer associated with the cloud provider environment. This reduces latency by allowing client devices to interact with servers located closer together, while also improving the security of resources within the cloud provider environment.

[0092] In at least one embodiment, such a system can be used to perform graphical rendering operations. In other embodiments, such a system can be used for other purposes, such as providing image or video content to test or validate autonomous machine applications, or to perform deep learning operations. In at least one embodiment, such a system can be implemented using an edge device or can include one or more virtual machines (VMs). In at least one embodiment, such a system can be implemented at least partially in a data center or at least partially using cloud computing resources. Inference and training logic

[0093] Fig. 7A Inference and / or Training Logic 715 is illustrated, which is used to perform inference and / or training operations associated with one or more embodiments. Details regarding Inference and / or Training Logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided.

[0094] In at least one embodiment, inference and / or training logic 715 can, without limitation, include code and / or data storage 701 for storing forward and / or output weights and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, training logic 715 can include or be coupled to code and / or data storage 701 to store graph code or other software for controlling timing and / or sequencing, into which weight and / or other parameter information is to be loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, code and / or data memory 701 stores weight parameters and / or input / output data of each layer of a neural network, which, in conjunction with one or more embodiments, are trained or used during the forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data memory 701 may be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache memory or system memory of a processor.

[0095] In at least one embodiment, any section of code and / or data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data memory 701 can be a cache memory, a dynamic randomly addressable memory (“DRAM”), a static randomly addressable memory (“SRAM”), a non-volatile memory (e.g., flash memory), or another type of memory.In at least one embodiment, the choice of whether code and / or data storage 701 is, for example, internal or external to a processor, or consists of DRAM, SRAM, Flash or another type of memory, may depend on available on-chip versus off-chip memory, latency requirements of trained and / or inferencing functions being performed, batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0096] In at least one embodiment, inference and / or training logic 715 can, without limitation, include a code and / or data store 705 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, code and / or data store 705 stores weight parameters and / or input / output data of each layer of a neural network that are trained or used in conjunction with one or more embodiments during the backward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, training logic 715 can include or be coupled to code and / or data memory 705 to store graph code or other software for controlling timing and / or sequencing, into which weight and / or other parameter information is to be loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, any portion of code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache memory or system memory of a processor.In at least one embodiment, any section of code and / or data memory 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data memory 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the choice of whether code and / or data memory 705 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash memory, or another type of memory, can depend on available on-chip versus off-chip memory, latency requirements of trained and / or inferencing functions being performed, batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0097] In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 can be separate memory structures. In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 can be the same memory structure. In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any portion of code and / or data memory 701 and code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache memory or system memory of a processor.

[0098] In at least one embodiment, inference and / or training logic 715 can, without limitation, include one or more arithmetic logic unit(s) (“ALU(s)”) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations that are at least partially based on or specified by training and / or inference code (e.g., graph code), wherein a result thereof can generate activations (e.g., output values ​​of layers or neurons within a neural network) that are stored in an activation memory 720, which are functions of input / output and / or weight parameter data that are stored in code and / or data memory 701 and / or code and / or data memory 705.In at least one embodiment, activations stored in activation memory 720 are generated according to linear algebraic and / or matrix-based mathematics performed by ALU(s) 710 in response to the execution of instructions or other code, wherein weight values ​​stored in code and / or data memory 705 and / or code and / or data memory 701 are used as operands together with other values, such as deviation values, gradient information, pulse values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data memory 705 or code and / or data memory 701 or in another on-chip or off-chip memory.

[0099] In at least one embodiment, ALU(s) 710 are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, ALU(s) 710 may be contained within the execution units of a processor or otherwise in a bank of ALUs that the execution units of a processor access either within the same processor or that are distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed functional units, etc.).In at least one embodiment, code and / or data memory 701, code and / or data memory 705, and activation memory 720 can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they can be located on different processors or other hardware logic devices or circuits, or a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation memory 720 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache memory or system memory of a processor.Furthermore, inference and / or training code may be stored with other code that a processor or other hardware logic or circuitry can access and that is retrieved and / or processed using the fetch, decode, schedule, execute, retract and / or other logical circuitry of a processor.

[0100] In at least one embodiment, activation memory 720 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, activation memory 720 can be located wholly or partially inside or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether activation memory 720 is, for example, internal or external to a processor, or whether it consists of DRAM, SRAM, flash, or another type of memory, can depend on available on-chip versus off-chip memory, latency requirements of trained and / or inferencing functions being performed, batch size of data used in the inference and / or training of a neural network, or a combination of these factors. In at least one embodiment, inference and / or training logic 715, which is implemented in Fig. Figure 7A illustrates the use of an application-specific integrated circuit (“ASIC”), such as Google’s Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, inference and / or training logic 715, which is described in Figure 7A, can be used in conjunction with ... Fig. 7A illustrates how they can be used in conjunction with hardware of a central processing unit (“CPU”), hardware of a graphics processing unit (“GPU”) or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0101] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one or more embodiments. In at least one embodiment, inference and / or training logic 715 can, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and / or training logic 715, which is described in Fig. Figure 7B illustrates this, in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, inference and / or training logic 715, which is described in Figure 7B, can be used. Fig. Figure 7B illustrates the use of inference and / or training logic 715 in conjunction with hardware of a central processing unit (CPU), hardware of a graphics processing unit (GPU), or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, inference and / or training logic 715 includes, without limitation, code and / or data memory 701 and code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including deviation values, gradient information, impulse values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Figure 7B, the inference and / or training logic 715 includes code and / or data memory 701 and code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including deviation values, gradient information, impulse values, and / or other parameter or hyperparameter information. Fig. As illustrated in Figure 7B, each of code and / or data memory 701 and code and / or data memory 705 is assigned to a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and / or data memory 701 and code and / or data memory 705, respectively, with one result of these functions being stored in activation memory 720.

[0102] In at least one embodiment, each of the code and / or data storage devices 701 and 705, or corresponding computing hardware 702 and 706, corresponds to different layers of a neural network, such that the resulting activation from a "memory / computing pair 701 / 702" of code and / or data storage device 701 and computing hardware 702 is provided as an input for a "memory / computing pair 705 / 706" of code and / or data storage device 705 and computing hardware 706, in order to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included after or in parallel to memory / computing pairs 701 / 702 and 705 / 706 in inference and / or training logic 715. DATA CENTER

[0103] Fig. Figure 8 illustrates an exemplary data center 800 in which at least one embodiment can be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0104] In at least one embodiment, as in Fig. As shown in Figure 8, the data center infrastructure layer 810 can include a resource orchestrator 812, clustered compute resources 814, and node compute resources (“node CRs”) 816(1)-816(N), where “N” is any positive integer. In at least one embodiment, the node CRs 816(1)-816(N) can include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units, etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output devices (“NW I / O” devices), network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more node CRs can be made from the node CRss 816(1)-816(N) be a server that has one or more of the above-mentioned computing resources.

[0105] In at least one embodiment, the grouped compute resources 814 can include separate groupings of node CRs located in one or more racks (not shown), or many racks located in data centers at different geographic locations (also not shown). Separate groupings of node CRs within the grouped compute resources 814 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs containing CPUs or processors can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power modules, cooling modules and network switches in any combination.

[0106] In at least one embodiment, the resource orchestrator 812 can configure or otherwise control one or more node CRs 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 812 can include a software design infrastructure (“SDI”) management entity for the data center 800. In at least one embodiment, the resource orchestrator 812 can include hardware, software, or a combination thereof.

[0107] In at least one embodiment, as in Fig. As shown in Figure 8, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 can include a framework to support software 832 of software layer 830 and / or one or more application(s) 842 of application layer 840. In at least one embodiment, software 832 or application(s) 842 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter “Spark”), which can use, but is not limited to, the distributed file system 828 for large-scale data processing (e.g., “big data”).In at least one embodiment, the job scheduler 822 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 800. In at least one embodiment, the configuration manager 824 can be able to configure different layers, such as the software layer 830 and the framework layer 820, which includes Spark and the distributed file system 828, to support large-scale data processing. In at least one embodiment, the resource manager 826 can be able to manage clustered or grouped compute resources that are mapped to or allocated to support the distributed file system 828 and the job scheduler 822. In at least one embodiment, clustered or grouped compute resources can include the grouped compute resource 814 on the data center infrastructure layer 810.In at least one embodiment, the resource manager 826 can coordinate with the resource orchestrator 812 to manage these mapped or allocated computing resources.

[0108] In at least one embodiment, the software 832, which is included in software layer 830, may include software used by at least sections of the node CRs 816(1)-816(N), the grouped computing resources 814, and / or the distributed file system 828 of framework layer 820. The one or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0109] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by at least sections of the node compute resources 816(1)-816(N), the grouped compute resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may include, but are not limited to, any number of a genomic application, a cognitive compute application, and a machine learning application, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0110] In at least one embodiment, any configuration manager 824, resource manager 826, and resource orchestrator 812 can implement any number and type of self-modifying actions based on any set and type of data acquired in any technically feasible way. In at least one embodiment, self-modifying actions can relieve a data center operator of data center 800 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing sections of a data center.

[0111] In at least one embodiment, the Data Center 800 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and computing resources described above in relation to the Data Center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above in relation to the Data Center 800, by using weight parameters calculated by one or more training techniques described herein.

[0112] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform inference from information, such as image recognition, speech recognition, or other artificial intelligence services.

[0113] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided. In at least one embodiment, inference and / or training logic 715 can be integrated into system 715. Fig. 8. used to infer or predict operations that are based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described herein.

[0114] Such components can be used for animation systems. COMPUTER SYSTEMS

[0115] Fig. 9 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof, 900, formed with a processor that may include execution units for carrying out an instruction, according to at least one embodiment. In at least one embodiment, the computer system 900 may, without limitation, include a component, such as a processor 902, to employ execution units that include logic for carrying out algorithms for processing data, according to the present disclosure, as in the embodiment described herein.In at least one embodiment, the Computer System 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation in Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, and the like) may also be used. In at least one embodiment, the Computer System 900 may run a version of the WINDOWS operating system available from Microsoft Corporation in Redmond, California, although other operating systems (for example, UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0116] Embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip, network computers (“NetPCs”), set-top boxes, network hubs, wide-area network (“WAN”) switches, or any other system capable of executing one or more instructions according to at least one embodiment.

[0117] In at least one embodiment, the computer system 900 can, without limitation, include a processor 902, which can, without limitation, include one or more execution units 908 for performing machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 can be a multi-processor system. In at least one embodiment, the processor 902 can, without limitation, include a microprocessor for computing complex instruction sets (“CISC”), a microprocessor for computing reduced instruction sets (“RISC”), a microprocessor for computing very long instruction words (“VLIW”), a processor implementing a combination of instruction sets, or any other processing device, such as, for example, a digital signal processor.In at least one embodiment, the processor 902 can be coupled to a processor bus 910, which can transmit data signals between the processor 902 and other components in the computer system 900.

[0118] In at least one embodiment, the processor 902 can, without limitation, include an internal Level 1 cache memory (“Cache”) 904 (“L1”). In at least one embodiment, the processor 902 can have a single internal cache or multiple levels of an internal cache. In at least one embodiment, the cache memory can be located outside the processor 902. Other embodiments can also include a combination of both internal and external caches, depending on a particular implementation and specific requirements. In at least one embodiment, the register file 906 can store different types of data in different registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0119] In at least one embodiment, the execution unit 908, including without limitation logic for performing integer and floating-point operations, is also located in the processor 902. In at least one embodiment, the processor 902 may also include a read-only memory (“ROM”) containing microcode (“ucode”) that stores microcode for specific macro instructions. In at least one embodiment, the execution unit 908 may include logic for handling a packed instruction set 909. In at least one embodiment, by including the packed instruction set 909 in an instruction set of a general-purpose processor 902, together with associated circuitry for executing instructions, operations used by many multimedia applications can be performed using packed data in a general-purpose processor 902.In one or more embodiments, many multimedia applications can be accelerated and run more efficiently by using the full width of a processor's data bus to perform operations on packed data, which can eliminate the need to transfer smaller data units across the processor's data bus to perform one or more operations on one data element at a time.

[0120] In at least one embodiment, the execution unit 908 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 900 can include a memory 920 without restriction. In at least one embodiment, the memory 920 can be implemented as a dynamic random-access memory (“DRAM”) device, a static random-access memory (“SRAM”) device, a flash memory device, or another storage device. In at least one embodiment, the memory 920 can store instructions 919 and / or data 921 represented by data signals that can be executed by the processor 902.

[0121] In at least one embodiment, the system logic chip can be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip can include, without restriction, a memory controller hub (“MCH”) 916, and the processor 902 can communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 can provide a high-bandwidth memory path 918 to the memory 920 for instruction and data storage and for storing graphics instructions, data, and textures. In at least one embodiment, the MCH 916 can route data signals between the processor 902, the memory 920, and other components in the computer system 900 and bridge data signals between the processor bus 910, the memory 920, and a system I / O 922. In at least one embodiment, the system logic chip can provide a graphics port for coupling with a graphics controller.In at least one embodiment, the MCH 916 can be coupled to the memory 920 by a high-bandwidth memory path 918, and the graphics / video card 912 can be coupled to the MCH 916 by an Accelerated Graphics Port Connection (“AGP”) 914.

[0122] In at least one embodiment, the computer system 900 can use the system I / O 922, which is a proprietary hub interface bus, to couple the MCH 916 to the I / O controller hub (“ICH”) 930. In at least one embodiment, the ICH 930 can provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can, without limitation, include a high-speed I / O bus for connecting peripheral devices to the memory 920, the chipset, and the processor 902. Examples can include, without limitation, an audio controller 929, a firmware hub (“Flash BIOS”) 928, a wireless transceiver 926, a data storage device 924, a legacy I / O controller 923 containing user input and keyboard interfaces 925, a serial expansion port 927, such as a universal serial bus (“USB”), and a network controller 934.The 924 data storage device can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or another mass storage device.

[0123] Illustrated in at least one embodiment Fig. 9 a system that includes interconnected hardware devices or “chips”, whereas Fig. Figure 9 illustrates an exemplary system-on-a-chip (“SoC”) in other embodiments. In at least one embodiment, devices can be connected using proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the Computer System 900 are interconnected using Compute Express Link (CXL) connections.

[0124] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided. In at least one embodiment, inference and / or training logic 715 can be integrated into system 715. Fig. 9. used to infer or predict operations that are based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described herein.

[0125] Such components can be used for animation systems.

[0126] Fig. Figure 10 is a block diagram illustrating an electronic device 1000 for using a processor 1010, according to at least one embodiment. In at least one embodiment, the electronic device 1000 can be, for example, without limitation, a notebook, a mast server, a rack server, a blade server, a laptop, a desktop computer, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0127] In at least one embodiment, the system 1000 can include, without restriction, the processor 1010, which is communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1010 is coupled using a bus or interface, such as a 1°C bus, a system management bus (“SM bus”), a low-pin-count bus (LPC), a serial peripheral interface (“SPI”), a high-resolution audio bus (“HDA”), an advanced technology serial port bus (“SATA”), a universal serial bus (“USB”) (versions 1, 2, 3), or a universal asynchronous receiver / transmitter bus (“UART”). In at least one embodiment, the following is illustrated: Fig. 10 a system that includes interconnected hardware devices or “chips”, whereas Fig. 10. In other embodiments, an exemplary system-on-a-chip (“SoC”) can be illustrated. In at least one embodiment, in Fig. 10 illustrated devices can be connected using proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of Fig. 10 are interconnected using Compute Express Link (CXL) connections.

[0128] In at least one embodiment, Fig. 10 a display 1024, a touchscreen 1025, a touchpad 1030, a near field communication (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, an Express chipset (“EC”) 1035, a trusted platform module (“TPM”) 1038, BIOS / firmware / flash memory (“BIOS, FW Flash”) 1022, a DSP 1060, a drive 1020, such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”), a wireless local area network (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide area network (“WWAN”) 1056, a global positioning system (GPS) 1055, a camera (“USB 3.0 camera”) 1054, such as a The USB 3.0 camera and / or a low double data rate (“LPDDR”) (“LPDDR3”) 1015 memory unit, implemented, for example, in the LPDDR3 standard, are included. These components can each be implemented in any suitable way.

[0129] In at least one embodiment, other components can be communicatively coupled to the processor 1010 via the components discussed above. In at least one embodiment, an accelerometer 1041, an ambient light sensor (“ALS”) 1042, a compass 1043, and a gyroscope 1044 can be communicatively coupled to the sensor hub 1040. In at least one embodiment, a thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 can be communicatively coupled to the EC 1035. In at least one embodiment, loudspeakers 1063, headphones 1064, and a microphone (“mic”) 1065 can be communicatively coupled to an audio unit (“audio codec and Class-D amplifier”) 1062, which in turn can be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1062 can, for example and without limitation, include an audio encoder / decoder (“codec”) and a class-D amplifier.In at least one embodiment, the SIM card (“SIM”) 1057 can be communicatively coupled with the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, can be implemented in a next-generation form factor (“NGFF”).

[0130] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided. In at least one embodiment, inference and / or training logic 715 can be integrated into system 715. Fig. 10. used to infer or predict operations that are based at least partially on weight parameters calculated using neural network training operations, neural network functions and / or neural network architectures or neural network use cases described herein.

[0131] Such components can be used for animation systems.

[0132] Fig. Figure 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, the system 1100 includes one or more processor(s) 1102 and one or more graphics processor(s) 1108 and can be a single-processor desktop system, a multi-processor workstation system, or a server system with a large number of processor(s) 1102 or processor core(s) 1107. In at least one embodiment, the system 1100 is a processing platform integrated into an integrated system-on-a-chip (SoC) circuit for use in mobile, handheld, or embedded devices.

[0133] In at least one embodiment, the system 1100 can include a game console, including a game and media console, a mobile game console, a handheld game console, or an online game console, or be integrated into a server-based game platform. In at least one embodiment, the system 1100 is a mobile phone, a smartphone, a tablet computer, or a mobile internet device. In at least one embodiment, the processing system 1100 can also include, be coupled to, or be integrated with a wearable device, such as a wearable smartwatch device, smart glasses device, augmented reality device, or virtual reality device.In at least one embodiment, the processing system 1100 is a television or set-top box device with one or more processor(s) 1102 and a graphical user interface generated by one or more graphics processor(s) 1108.

[0134] In at least one embodiment, one or more processor(s) 1102 each include one or more processor core(s) 1107 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor core(s) 1107 is configured to process a specific instruction set 1109. In at least one embodiment, the instruction set 1109 can facilitate the computation of complex instruction sets (CISC), reduced instruction sets (RISC), or computation via a very long instruction word (VLIW). In at least one embodiment, the processor core(s) 1107 can each process a different instruction set 1109, which may include instructions to facilitate the emulation of other instruction sets.In at least one embodiment, the processor core(s) 1107 may also include other processing devices, such as a digital signal processor (DSP).

[0135] In at least one embodiment, the processor(s) 1102 includes a cache memory 1104. In at least one embodiment, the processor(s) 1102 may have a single internal cache or multiple levels of an internal cache. In at least one embodiment, the cache memory is shared by different components of the processor(s) 1102. In at least one embodiment, the processor(s) 1102 also uses an external cache (e.g., a Level 3 cache (L3) or a Level 10 cache (LLC)) (not shown), which can be shared by the processor core(s) 1107 using known cache coherence techniques. In at least one embodiment, the register file 1106 is additionally included in the processor(s) 1102, which may include different types of registers for storing different types of data (e.g.,Integer registers, floating-point registers, status registers, and an instruction pointer register). In at least one embodiment, the register file may contain 1106 universal registers or other registers.

[0136] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 to transmit communication signals, such as address, data, or control signals, between the processor(s) 1102 and other components in the system 1100. In at least one embodiment, the interface bus(s) 1110 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus(s) 1110 is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor(s) 1102 includes an integrated memory controller 1116 and a platform controller hub 1130.In at least one embodiment, the storage controller 1116 facilitates communication between a storage device and other components of the system 1100, while the platform controller hub (PCH) 1130 provides connections to I / O devices via a local I / O bus.

[0137] In at least one embodiment, the storage device 1120 can be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or another storage device with suitable performance to serve as process memory. In at least one embodiment, the storage device 1120 can operate as system memory for the system 1100 to store data 1122 and instructions 1121 for use when one or more processor(s) 1102 are executing an application or process. In at least one embodiment, the memory controller 1116 also couples with an optional external graphics processor 1112, which can communicate with one or more graphics processor(s) 1108 within the processor(s) 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 can connect to the processor(s) 1102. In at least one embodiment, the display device 1111 can include one or more internal display devices, such as in a mobile electronic device or a laptop device, or external display devices connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 can include a head-mounted display (HMD), such as a stereoscopic display device, for use in virtual reality (VR) or augmented reality (AR) applications.

[0138] In at least one embodiment, the platform controller hub 1130 enables peripheral devices to connect to the storage device 1120 and the processor(s) 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can connect via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors can include 1125 touchscreen sensors, pressure sensors or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with system firmware and can, for example, be a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) couples with the interface bus(s) 1110. In at least one embodiment, the audio controller 1146 is a multi-channel, high-resolution audio controller. In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling legacy (e.g.,Personal System 2 (PS / 2) devices with the system. In at least one embodiment, the platform controller hub 1130 can also connect to one or more Universal Serial Bus (USB) controller(s) 1142, which connect input devices such as keyboard and mouse combinations 1143, a camera 1144, or other USB input devices.

[0139] In at least one embodiment, an instance of the memory controller 1116 and the platform controller hub 1130 can be integrated into a discrete external graphics processor, such as the external graphics processor 1112. In at least one embodiment, the platform controller hub 1130 and / or the memory controller 1116 can be located outside of one or more processor(s) 1102. For example, in at least one embodiment, the system 1100 can include an external memory controller 1116 and platform controller hub 1130, which can be configured as a memory controller hub and peripheral controller hub within a system chipset that communicates with the processor(s) 1102.

[0140] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided. In at least one embodiment, sections or all of the inference and / or training logic 715 can be incorporated into the graphics processor 1500. For example, in at least one embodiment, training and / or inference techniques described herein can utilize one or more ALUs embodied in a graphics processor. Furthermore, in at least one embodiment, inference and / or training operations described herein can be performed using logic other than that provided in Fig. 7A and / or Fig. The logic illustrated in Figure 7B can be carried out. In at least one embodiment, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of a graphics processor to perform one or more of the machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0141] Such components can be used for animation systems.

[0142] Fig. Figure 12 is a block diagram of a processor 1200 comprising one or more processor core(s) 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to and including the additional core 1202N, which are represented by boxes with dashed lines. In at least one embodiment, each of the processor core(s) 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0143] In at least one embodiment, the internal cache unit(s) 1204A-1204N and the shared cache unit(s) 1206 constitute a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache unit(s) 1204A-1204N may include at least one level of an instruction and data cache within each processor core and one or more levels of a shared middle-level cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, with a highest cache level prior to external memory being classified as an LLC. In at least one embodiment, cache coherence logic maintains coherence between different cache units 1206 and 1204A-1204N.

[0144] In at least one embodiment, the processor 1200 can also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functionality for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external storage devices (not shown).

[0145] In at least one embodiment, one or more of the processor core(s) 1202A-1202N include support for simultaneous multi-threading. In at least one embodiment, the system agent core 1210 includes components for coordinating the processor core(s) 1202A-1202N during multi-threading processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU) that includes logic and components for controlling one or more power states of the processor core(s) 1202A-1202N and the graphics processor 1208.

[0146] In at least one embodiment, the processor 1200 additionally includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled to one or more shared cache units 1206 and the system agent core 1210, which includes one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a display controller 1211 for driving graphics processor output to one or more coupled displays. In at least one embodiment, the display controller 1211 can also be a separate module coupled to the graphics processor 1208 via at least one connection, or it can be integrated into the graphics processor 1208.

[0147] In at least one embodiment, a ring-based interconnect 1212 is used to couple internal components of the processor 1200. In at least one embodiment, an alternative interconnect can be used, such as a point-to-point connection, a switched connection, or other techniques. In at least one embodiment, the graphics processor 1208 couples to a ring-based interconnect 1212 via an I / O connection 1213.

[0148] In at least one embodiment, the I / O connection 1213 represents at least one of several I / O connection variants, including an on-package I / O connection, which facilitates communication between different processor components and a high-performance embedded memory module 1218, such as an eDRAM module. In at least one embodiment, each of the processor core(s) 1202A-1202N and the graphics processor 1208 use embedded memory modules 1218 as a shared last-level cache.

[0149] In at least one embodiment, the processor core(s) 1202A - 1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the processor core(s) 1202A - 1202N are heterogeneous with respect to the instruction set architecture (ISA), wherein one or more of the processor core(s) 1202A - 1202N execute a common instruction set, while one or more other cores of the processor core(s) 1202A - 1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor core(s) 1202A - 1202N are heterogeneous with respect to the microarchitecture, wherein one or more cores with a relatively higher power consumption are coupled with one or more power cores with a lower power consumption.In at least one embodiment, the 1200 processor can be implemented on one or more chips or as an integrated SoC circuit.

[0150] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are given below in conjunction with Fig. 7A and / or Fig. 7B provided. In at least one embodiment, sections or all of the inference and / or training logic 715 can be incorporated into the processor 1200. For example, in at least one embodiment, training and / or inference techniques described herein can use one or more of the ALUs located in the graphics processor 1208, the graphics core(s) 1202A-1202N, or other components in Fig. 12 are embodied. Furthermore, in at least one embodiment, the inference and / or training operations described herein can be performed using logic other than that described in Fig. 7A and / or Fig. The logic illustrated in Figure 7B can be carried out. In at least one embodiment, weight parameters can be stored in an on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of the 1200 graphics processor to execute one or more of the machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0151] Such components can be used for animation systems. VIRTUALIZED COMPUTING PLATFORM

[0152] Fig. Figure 13 is an exemplary data flow diagram for a process 1300 for generating and deploying an image processing and inference pipeline, according to at least one embodiment. In at least one embodiment, the process 1300 can be used with imaging devices, processing devices, and / or other types of devices on one or more facilities 1302. The process 1300 can be executed within a training system 1304 and / or a deployment system 1306. In at least one embodiment, the training system 1304 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object recognition algorithms, computer vision algorithms, etc.) for use in the deployment system 1306.In at least one embodiment, the deployment system 1306 can be configured to offload processing and computing resources to a distributed computing environment in order to reduce infrastructure requirements at the facility 1302. In at least one embodiment, one or more applications in a pipeline can use or call services (e.g., inference, visualization, computation, AI, etc.) of the deployment system 1306 during application execution.

[0153] In at least one embodiment, some of the applications used in advanced processing and inference pipelines can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models can be trained at the facility 1302 using data 1308 (such as imaging data) generated at the facility 1302 (and stored on one or more Picture Archiving and Communication System (PACS) servers at the facility 1302), can be trained using imaging or sequencing data 1308 from another facility (other facilities), or a combination thereof.In at least one embodiment, the training system 1304 can be used to provide applications, services and / or other resources for generating functional, deployable machine learning models for the deployment system 1306.

[0154] In at least one embodiment, the model registry 1324 can be supported by object storage that can support versioning and object metadata. In at least one embodiment, object storage can be accessed, for example, from within a cloud platform via a cloud storage-compatible application programming interface (API). In at least one embodiment, machine learning models within the model registry 1324 can be uploaded, listed, modified, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications, so that models can be executed as part of the execution of containerized instantiations of applications.

[0155] In at least one embodiment, the training system 1304 ( Fig. 13) include a scenario in which the facility 1302 trains its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by imaging device(s), sequencing devices, and / or other device types can be received. In at least one embodiment, once the imaging data 1308 is received, AI-assisted annotation 1310 can be used to assist in generating annotations that correspond to the imaging data 1308 to be used as ground-truth data for a machine learning model. In at least one embodiment, AI-assisted annotation 1310 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations that correspond to certain types of imaging data 1308 (e.g.,(of certain devices). In at least one embodiment, AI-assisted annotation 1310 can then be used directly or can be adapted or fine-tuned using an annotation tool to generate ground-truth data. In at least one embodiment, AI-assisted annotation 1310, labeled data 1312, or a combination thereof can be used as ground-truth data for training a machine learning model. In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and can be used by the deployment system 1306 as described herein.

[0156] In at least one embodiment, a training pipeline may include a scenario in which the facility 1302 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry 1324. In at least one embodiment, the model registry 1324 may contain machine learning models that are trained to perform a variety of different inference tasks on imaging data.In at least one embodiment, machine learning models in model registration 1324 may have been trained on imaging data from facilities other than facility 1302 (e.g., remotely located facilities). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, if training is performed on imaging data from a specific location, it may take place at that location or at least in a manner that protects the confidentiality of imaging data or restricts the transmission of imaging data outside of premises. In at least one embodiment, once a model has been trained—or partially trained—at a location, a machine learning model may be added to model registration 1324.In at least one embodiment, a machine learning model can then be retrained or updated on any number of other facilities, and a retrained or updated model can be made available in the model registry 1324. In at least one embodiment, a machine learning model can then be selected from the model registry 1324—and designated as output model(s) 1316—and can be used in the deployment system 1306 to perform one or more processing tasks for one or more applications of a deployment system.

[0157] In at least one embodiment, a scenario may involve that the facility 1302 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model registry 1324 may not be fine-tuned or optimized for imaging data 1308 generated at the facility 1302 due to differences in the population, the robustness of training data used to train a machine learning model, the variety of anomalies in training data, and / or other problems with training data.In at least one embodiment, AI-assisted annotation 1310 can be used to assist in generating annotations corresponding to the imaging data 1308, which are to be used as ground-truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 can be used as ground-truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model can be referred to as model training 1314. In at least one embodiment, model training 1314—e.g., AI-assisted annotation 1310, labeled data 1312, or a combination thereof—can be used as ground-truth data for retraining or updating a machine learning model.In at least one embodiment, a trained machine learning model can be designated as output model(s) 1316 and can be used by the deployment system 1306 as described herein.

[0158] In at least one embodiment, the deployment system 1306 can include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, the deployment system 1306 can include a software "stack" such that software 1318 can be built on top of services 1320 and use services 1320 to perform some or all of the processing tasks, and services 1320 and software 1318 can be built on top of hardware 1322 and use hardware 1322 to perform the processing, storage, and / or other computational tasks of the deployment system 1306. In at least one embodiment, software 1318 can include any number of different containers, each container being capable of instantiating an application. In at least one embodiment, each application can perform one or more processing tasks in an advanced processing and inference pipeline (e.g.,Inference, object recognition, feature recognition, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline can be defined based on selections of different containers that are desired or required for processing imaging data 1308, in addition to containers that receive and configure imaging data for use by each container and / or for use by facility 1302 after processing by a pipeline (e.g., to convert outputs back into a usable data type). In at least one embodiment, a combination of containers within software 1318 (e.g.,a pipeline) is referred to as a virtual instrument (as described in more detail herein), and a virtual instrument can use services 1320 and hardware 1322 to perform some or all of the processing tasks of applications instantiated in containers.

[0159] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, input data can represent one or more images, video, and / or other data representations generated by one or more imaging devices. In at least one embodiment, data can be preprocessed as part of the data processing pipeline to prepare it for processing by one or more applications.In at least one embodiment, post-processing can be performed on an output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or deployed neural networks, which may include one or more output models 1316 of the training system 1304.

[0160] In at least one embodiment, tasks of the data processing pipeline can be encapsulated in one or more containers, each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published in a private (e.g., access-restricted) area of ​​a container registry (described in more detail here), and trained or deployed models can be stored in the model registry 1324 and associated with one or more applications. In at least one embodiment, images of applications (e.g.,Container images) are available in a container registry, and once they have been selected by a user from a container registry for insertion into a pipeline, an image can be used to create a container for instantiating an application for use by a user's system.

[0161] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) can develop, publish, and store applications (e.g., as containers) for performing image processing and / or inference on supplied data. In at least one embodiment, development, publication, and / or storage can be performed using a software development kit (SDK) that is linked to a system (e.g., to ensure that an application and / or a developed container is compliant or compatible with a system). In at least one embodiment, an application under development can be tested locally (e.g., at a first facility, on data from a first facility) using an SDK that includes at least some of the services 1320 as a system (e.g., System 1200 from Fig. 12) can support. Since DICOM objects can contain anywhere from one to hundreds of images or other data types, and due to variation in data, a developer in at least one embodiment may be responsible for managing (e.g., setting constructs for, building preprocessing in an application, etc.) the extraction and preparation of incoming data. In at least one embodiment, once validated by System 1300 (e.g., for accuracy), an application in a container registry may be available for selection and / or implementation by a user to perform one or more processing tasks with respect to data at a user's facility (e.g., a second facility).

[0162] In at least one embodiment, developers can then make applications or containers available over a network for access and use by users of a system (e.g., System 1300 from Fig. 13) share. In at least one embodiment, completed and validated applications or containers can be stored in a container registry, and linked machine learning models can be stored in the model registry 1324. In at least one embodiment, a requesting entity—providing an inference or image processing request—can search a container registry and / or model registry 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements for inclusion in a data processing pipeline, and submit an image processing request.In at least one embodiment, a request may include input data (and in some examples, linked patient data) necessary to execute a request, and / or may include a selection of application(s) and / or machine learning models to be executed when processing a request. In at least one embodiment, a request may then be directed to one or more components of the deployment system 1306 (e.g., a cloud) to perform processing in the data processing pipeline. In at least one embodiment, processing by the deployment system 1306 may involve referencing selected elements (e.g., applications, containers, models, etc.) from a container registry and / or model registry 1324. In at least one embodiment, once results are generated by a pipeline, results may be returned to a user for reference (e.g.,for viewing in a viewing application suite running on a local workstation or end device).

[0163] In at least one embodiment, services 1320 can be used to assist in processing or executing applications or containers in pipelines. In at least one embodiment, services 1320 can include compute services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 1320 can provide functionality common to one or more applications in software 1318, such that the functionality can be abstracted to a service that can be called or used by applications. In at least one embodiment, functionality provided by services 1320 can run dynamically and more efficiently while also scaling well by enabling applications to process data in parallel (e.g., using a parallel computing platform 1230). Fig. 12)). In at least one embodiment, services 1320 can be shared between and among different applications, instead of each application sharing the same functionality offered by services 1320 having to have its own instance of services 1320. In at least one embodiment, services can include an inference server or inference engine that can be used as non-restrictive examples to perform detection or segmentation tasks. In at least one embodiment, a model training service can be included that can provide capabilities for training and / or retraining machine learning models. In at least one embodiment, a data augmentation service can also be included that can provide extraction, resizing, scaling, and / or other augmentation of accelerated GPU data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.).In at least one embodiment, a visualization service can be used that can add image rendering effects—such as ray tracing, rasterization, denoising, sharpening, etc.—to add realism to two-dimensional (2D) and / or three-dimensional (3D) models. In at least one embodiment, virtual instrument services can be included that provide beam shaping, segmentation, inference, imaging, and / or support for other applications within virtual instrument pipelines.

[0164] In at least one embodiment, where services 1320 include an AI service (e.g., an inference service), one or more machine learning models can be executed by calling (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s) or process them as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can call an inference service to execute machine learning models to perform one or more processing operations associated with segmentation tasks.In at least one embodiment, software 1318, which implements an advanced processing and inference pipeline that includes a segmentation application and an anomaly detection application, can be streamlined because each application can call the same inference service to perform one or more inference tasks.

[0165] In at least one embodiment, the hardware 1322 may include GPUs, CPUs, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide efficient, purpose-built support for software 1318 and services 1320 in the deployment system 1306. In at least one embodiment, the use of GPU processing for local processing (e.g., in the facility 1302), within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 1306 may be implemented to improve the efficiency, accuracy, and effectiveness of image processing and generation.In at least one embodiment, software 1318 and / or services 1320 can be optimized for GPU processing with respect to deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least part of the computing environment of the deployment system 1306 and / or the training system 1304 can be run in a data center, one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, hardware 1322 can include any number of GPUs that can be called upon to perform parallel data processing as described herein. In at least one embodiment, the cloud platform can further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks.In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) can be run as a hardware abstraction and scaling platform using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems). In at least one embodiment, the cloud platform can integrate an application container clustering or orchestration system (e.g., Kubennetes) across multiple GPUs to enable seamless scaling and load balancing.

[0166] Fig. Figure 14 is a system diagram for an exemplary system 1400 for generating and inserting an imaging insertion pipeline, according to at least one embodiment. In at least one embodiment, system 1400 can be used to process process 1300 of Fig. 13 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, System 1400 may include Training System 1304 and Deployment System 1306. In at least one embodiment, Training System 1304 and Deployment System 1306 may be implemented using Software 1318, Services 1320 and / or Hardware 1322, as described herein.

[0167] In at least one embodiment, System 1400 (e.g., Training System 1304 and / or Deployment System 1306) can be implemented in a cloud computing environment (e.g., using Cloud 1426). In at least one embodiment, System 1400 can be implemented locally with respect to a healthcare facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in Cloud 1426 can be restricted to authorized users by means of security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carry appropriate authorization.In at least one embodiment, APIs of virtual instruments (described herein) or other instantiations of System 1400 may be restricted to a set of public IPs that have been audited or authorized for interaction.

[0168] In at least one embodiment, various components of System 1400 can communicate with each other using any of a variety of different network types, including, but not limited to, local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of System 1400 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can be carried out via data bus(s), wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0169] In at least one embodiment, the training system 1304 can execute training pipelines 1404 similar to those described herein with respect to Fig. 13 are described. In at least one embodiment, where one or more machine learning models are to be used in the deployment pipeline(s) 1410 by the deployment system 1306, the training pipelines 1404 can be used to train or retrain one or more (e.g., pre-trained) models and / or to implement one or more pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, output model(s) 1316 can be generated as a result of the training pipelines 1404. In at least one embodiment, the training pipelines 1404 can include any number of processing steps, such as, but not limited to, conversion or adaptation of imaging data (or other input data). In at least one embodiment, the training pipeline 1404, which is connected to a system with respect to Fig. Similar to the first example described in 13, the training pipeline 1404, which is similar to a first machine learning model, can be used for a first machine learning model. Fig. The second example described in 13 is similar, it can be used for a second machine learning model, and the training pipeline 1404, which is similar to a Fig. Similar to the third example described in 13, a third machine learning model can be used. In at least one embodiment, any combination of tasks within the training system 1304 can be used, depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for use, so that machine learning models may not be subjected to any processing by the training system 1304 and may be implemented by the deployment system 1306.

[0170] In at least one embodiment, the output model(s) 1316 and / or pretrained models 1406 may include any type of machine learning model, depending on the implementation or embodiment. In at least one embodiment, and without limitation, the machine learning models used by the system 1400 may include machine learning model(s) that employ linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbor (Knn), clustering (K), random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g., autocoders, convolutional, recurrent, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, fluid state machine, etc.), and / or other types of machine learning models.

[0171] In at least one embodiment, the training pipelines may include AI-supported annotation, as described herein with respect to at least Fig. 14 described in more detail. In at least one embodiment, the labeled data 1312 (e.g., conventional annotation) can be generated by any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or can be hand-drawn. In at least one embodiment, ground truth data can be synthetically produced (e.g., generated from computer models or renderings), truly produced (e.g., designed and produced from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), or human-annotated (e.g.,Labels or annotation expert, defines location of labels) and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data type used by machine learning models), corresponding ground-truth data generated by the training system 1304 may be available. In at least one embodiment, AI-assisted annotation may be performed as part of the deployment pipeline(s) 1410; either in addition to or instead of AI-assisted annotation included in the training pipelines 1404. In at least one embodiment, the system 1400 may include a multi-layered platform that may include a software layer (e.g., software 1318) of diagnostic applications (or other application types) capable of performing one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 may communicate with (e.g.,The system may be connected (via encrypted links) to PACS server networks of one or more institutions. In at least one embodiment, the System 1400 can be configured to access and reference data from PACS servers in order to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0172] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be called from one or more external environments (e.g., facility 1302). In at least one embodiment, applications can then call or execute one or more services 1320 to perform computational, AI, or visualization tasks assigned to the respective applications, and software 1318 and / or services 1320 can utilize hardware 1322 to perform processing tasks effectively and efficiently. In at least one embodiment, communications sent to or received from a training system 1304 and a deployment system 1306 can be made using a pair of DICOM adapters 1402A, 1402B.

[0173] In at least one embodiment, the deployment system 1306 can execute deployment pipeline(s) 1410. In at least one embodiment, the deployment pipeline(s) 1410 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomic devices, etc. – including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline 1410 for a single device can be designated as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.).In at least one embodiment, a single device may have more than one insertion pipeline 1410, depending on information desired from data generated by the device. In at least one embodiment, where anomaly detection from an MRI machine is desired, there may be a first insertion pipeline 1410, and where image enhancement from the output of an MRI machine is desired, there may be a second insertion pipeline 1410.

[0174] In at least one embodiment, an image generation application may include a processing task that involves the use of a machine learning model. In at least one embodiment, a user may wish to use their own machine learning model or select a machine learning model from model registration 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining constructs of applications, the use and implementation of applications for a particular user are presented as a more seamless user experience.In at least one embodiment, by incorporating other features of the system 1400 - such as services 1320 and hardware 1322 - the deployment pipeline(s) 1410 can be even more user-friendly, provide simpler integration and produce more accurate, efficient and timely results.

[0175] In at least one embodiment, the deployment system 1306 may include a user interface (“UI”) 1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in the deployment pipeline(s) 1410, to arrange applications, to modify or change applications or parameters or constructs thereof, to use and interact with the deployment pipeline(s) 1410 during setup and / or deployment, and / or to otherwise interact with the deployment system 1306. In at least one embodiment, although not illustrated with respect to the training system 1304, the UI 1414 (or another user interface) may be used to select models for use in the deployment system 1306, to select models for training or retraining in the training system 1304, and / or to otherwise interact with the training system 1304.

[0176] In at least one embodiment, the pipeline manager 1412 can be used in addition to an application orchestration system 1428 to manage interaction between applications or containers of the deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 can be configured to facilitate application-to-application, application-to-services 1320, and / or application-or-service-to-hardware 1322 interactions. In at least one embodiment, although illustrated as being contained in software 1318, this is not intended to be restrictive, and in some examples, the pipeline manager 1412 can be contained in services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, Docker, etc.) can be configured to facilitate application-to-services interactions, application-to-services interactions 1320, and / or application-to-services interactions 1322.) include a container orchestration system that can group applications as logical units for coordination, management, scaling, and deployment into containers. In at least one embodiment, each application can be run by assigning applications from the deployment pipeline(s) 1410 (e.g., a reconstruction application, a segmentation application, etc.) to individual containers in a self-contained environment (e.g., at a core level) to increase speed and efficiency.

[0177] In at least one embodiment, each application and / or container (or image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows users to focus on and attend to the tasks of a single application and / or container without being hindered by tasks of another application (or applications) or container (or containers). In at least one embodiment, communication and cooperation between different containers or applications can be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 1428 and / or the pipeline manager 1412 can facilitate communication and resource sharing between and between each of the applications or containers. In at least one embodiment, since one or more of the applications or containers in the deployment pipeline(s) 1410 can share the same services and resources, the application orchestration system 1428 can orchestrate, load balance, and determine the sharing of services or resources between and between different applications or containers.In at least one embodiment, a scheduler can be used to track resource requests from applications or containers, current or planned use of those resources, and resource availability. In at least one embodiment, a scheduler can thus allocate resources to different applications and distribute resources between and within applications with respect to system requirements and availability. In some examples, a scheduler (and / or another component of the application orchestration system 1428) can determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), the urgency of data output (e.g., to determine whether to perform real-time or delayed processing), and so on.

[0178] In at least one embodiment, services 1320, which are deployed and shared by applications or containers in the deployment system 1306, can include compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, and / or other service types. In at least one embodiment, applications can call (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, compute service(s) 1416 can be used by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 1416 can be used to perform parallel processing (e.g., using a parallel computing platform 1430) to process data by one or more applications and / or perform one or more tasks of a single application substantially simultaneously.In at least one embodiment, the parallel computing platform 1430 (e.g., NVIDIA's CUDA) can enable universal computing on GPUs (GPGPU) (e.g., GPUs / graphics 1422). In at least one embodiment, a software layer of the parallel computing platform 1430 can provide access to virtual instruction sets and parallel computing elements of GPUs for executing computing cores. In at least one embodiment, the parallel computing platform 1430 can include memory, and in some embodiments, memory can be shared between and among multiple containers and / or between and among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared memory segment of the Parallel Computing Platform 1430 (e.g., where several different stages of an application or multiple applications process the same information). In at least one embodiment, instead of creating a copy of data and moving data to different locations in memory (e.g., a read / write operation), the same data can be used at the same location in memory for any number of processing tasks (e.g., at the same time, at different times, etc.).In at least one embodiment, since data is used to generate new data as a result of processing, this information can be stored in a new data location and shared between different applications. In at least one embodiment, the data location and a location of updated or modified data can be part of a definition of how user data is understood within containers.

[0179] In at least one embodiment, AI service(s) 1418 can be used to perform inference services for executing machine learning model(s) associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI service(s) 1418 can utilize the AI ​​system 1424 to execute machine learning model(s) (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature recognition, classification, and / or other inference tasks. In at least one embodiment, applications of the insertion pipeline(s) 1410 can use one or more of the output model(s) 1316 from the training system 1304 and / or other application models to perform inference on imaging data. In at least one embodiment, two or more examples of inference using the application orchestration system 1428 (e.g.,a planner). In at least one embodiment, a first category may include a high-priority / low-latency path that can reach higher-level service agreements, such as for performing inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard priority path that can be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1428 can distribute resources (e.g., services 1320 and / or hardware 1322) based on priority paths for different inference tasks of AI service(s) 1418.

[0180] In at least one embodiment, a shared memory can be mounted on AI service(s) 1418 within the system 1400. In at least one embodiment, the shared memory can function as a cache (or another type of storage device) and can be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, the request can be received by a set of API instances of the deployment system 1306, and one or more instances can be selected (e.g., for best fit, load balancing, etc.) to process the request.In at least one embodiment, to process a request, a request can be entered into a database; a machine learning model can be ordered from the model registry 1324 if it is not already in a cache; a validation step can ensure that a suitable machine learning model is loaded into a cache (e.g., shared memory); and / or a copy of a model can be stored in a cache. In at least one embodiment, a scheduler (e.g., of the pipeline manager 1412) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server is not already started to execute a model, an inference server can be started.Any number of inference servers can be started per model. In at least one embodiment, a pull model where inference servers are clustered, models can be cached whenever load balancing is advantageous. In at least one embodiment, inference servers can be statically loaded onto corresponding distributed servers.

[0181] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally with multiple versions of a model). In at least one embodiment, if an instance of an inference server is not present when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, when an inference server is started, a model can be passed to an inference server, so that the same container can be used to serve different models, as long as the inference server is running as a separate instance.

[0182] In at least one embodiment, an inference request for a given application can be received during application execution, and a container (which, for example, hosts an instance of an inference server) can be loaded (if not already loaded), and a start procedure can be invoked. In at least one embodiment, preprocessing logic within a container can load, decode, and / or perform any additional preprocessing on incoming data (e.g., using a CPU(s) and / or GPU(s)). In at least one embodiment, once data is prepared for inference, a container can perform inference on the data as needed. In at least one embodiment, this can involve a single inference call on a single image (e.g., a hand X-ray) or it can require inference on hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize results before completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating a visualization, or generating text to summarize results. In at least one embodiment, different models or applications can be assigned different priorities. For example, some models may have a real-time priority (TAT < 1 min), while others may have a lower priority (e.g., TAT < 10 min). In at least one embodiment, model execution times can be measured by a requesting institution or entity and may include partner network throughput time as well as execution on an inference service.

[0183] In at least one embodiment, the transmission of requests between Services 1320 and inference applications can be hidden behind a software development kit (SDK), and robust transport can be provided through a queue. In at least one embodiment, a request is queued via an API for a single application / tenant ID combination, and an SDK pulls a request from the queue and submits it to an application. In at least one embodiment, a queue name can be provided in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication through a queue can be beneficial because it allows each instance of an application to begin work as it becomes available. Results can be pushed back through a queue to ensure that no data is lost.In at least one embodiment, queues can also provide the ability to segment work, since highest-priority work can go to a queue with the most instances of an associated application, while lowest-priority work can go to a queue with a single associated instance that processes tasks in a received order. In at least one embodiment, an application can run on a GPU-accelerated instance created in Cloud 1426, and an inference service can perform inference on a GPU.

[0184] In at least one embodiment, visualization service(s) 1420 can be used to generate visualizations for viewing outputs from applications and / or deployment pipeline(s) 1410. In at least one embodiment, GPUs / graphics 1422 can be used by visualization service(s) 1420 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, can be implemented by visualization service(s) 1420 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstruction, 2D tomography slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments can be used to provide a virtual interactive display or environment (e.g.,to create a virtual environment for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization service(s) 1420 may include an internal visualizer, kinetics, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0185] In at least one embodiment, Hardware 1322 can include GPUs / graphics 1422, AI system 1424, cloud 1426 and / or any other hardware used to run training system 1304 and / or deployment system 1306. In at least one embodiment, GPUs / graphics 1422 (e.g., NVIDIA TESLA and / or QUADRO GPUs) can include any number of GPUs that can be used to perform processing tasks of compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, other services and / or any features or functionality of software 1318. For example, with regard to AI service(s), 1418 GPUs / graphics 1422 can be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on outputs of machine learning models, and / or to perform inference (e.g., to run machine learning models).In at least one embodiment, Cloud 1426, AI System 1424, and / or other components of System 1400 can utilize GPUs / graphics 1422. In at least one embodiment, Cloud 1426 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI System 1424 can utilize GPUs, and Cloud 1426—or at least a section tasked with deep learning or inference—can be executed using one or more AI Systems 1424. Although Hardware 1322 is illustrated as discrete components, this is not intended to be restrictive, and any components of Hardware 1322 can be combined with or used by any other components of Hardware 1322.

[0186] In at least one embodiment, AI System 1424 may include a dedicated computing system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, AI System 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that can be run using a variety of GPUs / graphics 1422 in addition to CPUs, RAM, memory, and / or other components, features, or functionality. In at least one embodiment, one or more AI Systems 1424 may be deployed in cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of System 1400.

[0187] In at least one embodiment, Cloud 1426 can include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that can provide a GPU-optimized platform for performing processing tasks from System 1400. In at least one embodiment, Cloud 1426 can include an AI system 1424 for performing one or more AI-based tasks from System 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, Cloud 1426 can be integrated with an application orchestration system 1428 that utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 1320. In at least one embodiment, Cloud 1426 can be tasked with performing at least some of the services 1320 of System 1400, including compute service(s) 1416, AI service(s) 1418 and / or visualization service(s) 1420, as described herein.In at least one embodiment, Cloud 1426 can perform small and large batch inference (e.g., running NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run an application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality kinetics), and / or provide other functionality for System 1400.

[0188] Fig. Figure 15A illustrates a data flow diagram for a process 1500 for training, retraining, or updating a machine learning model, according to at least one embodiment. In at least one embodiment, process 1500 can be implemented using, as a non-limiting example, system 1400. Fig. 14. In at least one embodiment, Process 1500 can utilize services and / or hardware as described herein. In at least one embodiment, refined models 1512 generated by Process 1500 can be executed by a deployment system for one or more containerized applications in deployment pipelines.

[0189] In at least one embodiment, model training 1514 can involve retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as a customer record 1506, and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, output or loss layer(s) of the initial model 1504 can be reset, deleted, and / or replaced with updated or new output or loss layer(s). In at least one embodiment, the initial model 1504 can be modified using previously fine-tuned parameters (e.g.,weightings and / or biases) that remain from the previous training, so that the training or retraining 1514 may not take as long or require as much processing as training a model from scratch. In at least one embodiment, during model training 1514, by resetting or replacing the output or loss layer(s) of the initial model 1504, parameters for a new dataset can be updated and retuned based on loss calculations associated with the accuracy of the output or loss layer(s) when generating predictions about the new customer dataset 1506.

[0190] In at least one embodiment, pre-trained models 1506 may be stored in a data store or registry. In at least one embodiment, pre-trained models 1506 may have been trained, at least partially, at one or more facilities other than the facility executing process 1500. In at least one embodiment, to protect the privacy and rights of patients, subjects, or clients of different facilities, pre-trained models 1506 may have been trained on-site using customer or patient data generated on-site.In at least one embodiment, pre-trained models 1306 can be trained using a cloud and / or other hardware, but confidential, privacy-protected patient data may not be transferred to, used by, or accessed by any component of a cloud (or other hardware outside of premises). In at least one embodiment, if pre-trained models 1506 are trained using patient data from more than one facility, pre-trained models 1506 may have been trained individually for each facility before being trained on patient or customer data from another facility. In at least one embodiment, such as when a customer or patient data has been released for privacy reasons (e.g., by waiver, for experimental use, etc.),) or if customer or patient data is contained in a public dataset, customer or patient data is used by any number of facilities to train pre-trained models 1506 on-site and / or off-site, such as in a data center or other cloud computing infrastructure.

[0191] In at least one embodiment, when selecting applications for use in deployment pipelines, a user can also select machine learning models to be used for specific applications. In at least one embodiment, a user may not have a model available for use, allowing a user to select a pre-trained model for use with an application. In at least one embodiment, a pre-trained model may not be optimized to produce accurate results on a user's facility's customer dataset (e.g., based on patient diversity, demographics, types of medical imaging devices used, etc.). In at least one embodiment, before deploying a pre-trained model in a deployment pipeline for use with application(s), a pre-trained model may be updated, retrained, and / or fine-tuned for use at a particular facility.

[0192] In at least one embodiment, a user can select a pre-trained model to be updated, retrained, and / or fine-tuned, and this pre-trained model can be referred to as the initial model 1504 for a training system within the process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomic data, sequencing data, or other data types generated by devices on a facility) can be used to perform model training (which, without limitation, may include transfer learning) on ​​the initial model 1504 to produce a refined model 1512. In at least one embodiment, ground-truth data corresponding to the customer dataset 1506 can be generated by the training system 1304.In at least one embodiment, ground truth data can be generated at least partially by clinicians, scientists, doctors, and practitioners at an institution.

[0193] In at least one embodiment, AI-powered annotation can be used in some examples to generate ground-truth data. In at least one embodiment, AI-powered annotation (e.g., implemented using an AI-powered annotation SDK) can leverage machine learning models (e.g., neural networks) to generate suggested or predicted ground-truth data for a customer dataset. In at least one embodiment, a user can use annotation tools within a user interface (a graphical user interface (GUI)) on a computing device.

[0194] In at least one embodiment, the user 1510 can interact with a GUI via the computing device 1508 to edit or fine-tune (auto-)annotations. In at least one embodiment, a polygon editing feature can be used to move the vertices of a polygon to more accurate or fine-tuned locations.

[0195] In at least one embodiment, once the customer dataset 1506 has associated ground-truth data, ground-truth data (e.g., from AI-assisted annotation, manual labeling, etc.) can be used during model training to generate the refined model 1512. In at least one embodiment, the customer dataset 1506 can be applied to the initial model 1504 any number of times, and ground-truth data can be used to update parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 is generated, the refined model 1512 can be deployed within one or more deployment pipelines at a facility to perform one or more processing tasks related to medical imaging data.

[0196] In at least one embodiment, the refined model 1512 can be uploaded to pre-trained models in a model registry for selection by another facility. In at least one embodiment, this process can be completed at any number of facilities, allowing the refined model 1512 to be further refined any number of times on new datasets to generate a more universal model.

[0197] Fig. Figure 15B is an exemplary illustration of a client-server architecture 1532 for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment. In at least one embodiment, the AI-powered annotation tool 1536 can be instantiated based on a client-server architecture 1532. In at least one embodiment, the AI-powered annotation tool 1536 can assist radiologists in imaging applications, for example, in identifying organs and anomalies. In at least one embodiment, imaging applications can include software tools that assist the user 1510, as a non-restrictive example, in identifying some extreme points on a particular organ of interest in raw images 1534 (e.g., in a 3D MRI or CT scan) and automatically receiving annotated results for all 2D slices of a particular organ.In at least one embodiment, results can be stored in a data store as training data 1538 and used as (for example, and without limitation) ground-truth data for training. In at least one embodiment, if the computing device 1508 sends extreme points for AI-assisted annotation, a deep learning model can, for example, receive this data as input and return inference results of a segmented organ or anomaly. In at least one embodiment, pre-instantiated annotation tools, such as the AI-assisted annotation tool 1536, can be used. Fig.15B, can be improved by making API calls (e.g., API call 1544) to a server, such as an annotation support server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, an annotation model registry can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation on a specific organ or anomaly. These models can be further updated using training pipelines. In at least one embodiment, pre-installed annotation tools can be improved over time as new labeled data are added.

[0198] Different embodiments can be described by the following clauses: 1. A computer-implemented method comprising: Receiving audio data that corresponds to a spoken utterance; Calculate, using a transformer-based audio encoder and decoder, a weighted vector that specifies a variety of features associated with the audio data; Calculate, using the weighted vector and one or more component vectors, an animation vector corresponding to one or more positions for one or more feature points associated with a digital character representation; and Rendering the digital character representation based at least on the animation vector. 2. Computer-implemented method according to clause 1, wherein the transformer-based audio encoder is a pre-trained audio encoder, further comprising: Training the decoder at least based on the transformer-based audio encoder, whereby parameters for the transformer-based audio encoder are locked while the decoder is being trained. 3. Computer-implemented method according to clause 1, wherein the multitude of features is selected during training. 4. Computer-implemented method according to clause 1, further comprising: Receiving a respective layer vector, which is assigned to the multitude of features, for layers of the transformer-based audio encoder; Determining a layer weighting for individual layers; Applying the layer weighting to each individual layer; and determining the weighted vector. 5. Computer-implemented method according to clause 1, wherein the audio data has a duration that is less than a threshold duration. 6. Computer-implemented method according to clause 5, wherein the multitude of features is determined using transformer layers. 7. Computer-implemented method according to clause 1, wherein the one or more feature points correspond to at least one of facial features, a tongue position, an eye position or an extremity position. 8. Computer-implemented method according to clause 1, wherein the multitude of features is extracted from the audio data by processing using a convolutional neural network (CNN). 9. Computer-implemented method according to clause 1, wherein the component vector includes at least one emotion vector or style vector. 10. Computer-implemented method according to clause 1, wherein the decoder ignores information from one or more previous frames. 11. Computer-implemented procedure according to clause 10, further comprising: Penalizing movement between adjacent frames when the volume of the audio data is less than a volume threshold. 12. Processor, comprehensive: one or more processing units for: Calculate, using a transformer-based audio encoder and based on at least audio data corresponding to speech, a weighted feature vector associated with the audio data; Calculate, using the weighted feature vector and a component vector specifying one or more properties associated with the language, positional data for one or more feature points of one or more deformable body components of a virtual character; and Rendering, for one or more points in time in a sequence of points in time of the audio data, of image data representing the virtual character, based at least on the positional data, to create an animation of the character who appears to be speaking. 13. Processor according to clause 12, wherein the weighted feature vector is based at least on respective layer vectors for individual layers of the transformer-based audio encoder, wherein individual layer vectors are assigned to a plurality of features extracted from the audio data. 14. Processor according to clause 12, wherein parameters of the transformer-based audio encoder are locked during a training process for an associated decoder. 15. Processor according to clause 12, wherein the component vector includes at least one emotion vector or style vector. 16. Processor according to clause 12, wherein the processor is contained in at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed-reality (MR) content; a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative content operations using a Large Language Model (LLM); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for performing one or more generative content operations using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system that is implemented at least partially using cloud computing resources. 17. System, comprehensive: one or more processing units for generating an animation of a character using positional data representing one or more positions of one or more feature points of the character, wherein the positional data is calculated at least partially based on a transformer-based audio encoder processing audio data representing the speech, and component data specifying one or more values ​​corresponding to at least one of a style parameter or emotion parameter associated with the speech. 18. System according to clause 17, wherein the transformer-based audio encoder computes a weighted feature vector at least based on respective layer vectors for individual layers of the transformer-based audio encoder. 19. System according to clause 17, wherein parameters of the transformer-based audio encoder are locked during a training process for an associated decoder. 20. System according to clause 17, wherein the system comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed-reality (MR) content; a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative content operations using a Large Language Model (LLM); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for performing one or more generative content operations using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system that is implemented at least partially using cloud computing resources.

[0199] Other variations are within the scope of this disclosure. Although disclosed techniques are receptive to various modifications and alternative constructions, certain illustrated embodiments are shown in the drawings and have been described in detail above. It is understood, however, that the disclosure is not intended to be limited to any particular disclosed form or forms, but rather, on the contrary, to encompass all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the attached claims.

[0200] The use of the terms "a," "an," and "the" and similar references in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as a definition of a term. The terms "comprising," "having," "including," and "containing" are to be interpreted as open terms (meaning "including but not limited to") unless otherwise specified. The term "connected," when unmodified and referring to physical connections, is to be interpreted as partially or completely contained in, attached to, or connected to one another, even if something lies in between.The recitation of ranges of values ​​is herein intended merely as a short-lived procedure for individually referring to each separate value falling within the range, unless otherwise specified herein, and each separate value is included in the specification as if it were individually recited herein. The use of the term "set" (e.g., "a set of elements") or "subset" is, unless otherwise specified or the context contradicts, to be understood as a non-empty collection comprising one or more elements. Furthermore, unless otherwise specified or the context contradicts, the term "subset" of a corresponding set does not necessarily denote a proper subset of a corresponding set, but the subset and the corresponding set may be the same.

[0201] Subjunctive language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless explicitly stated otherwise or the context clearly contradicts it, is otherwise understood in relation to the context as it is generally used to indicate that an element, term, etc., can be either A, B, C, or any non-empty subset of the set of A, B, and C. For example, in an illustrative example of a three-element set, subjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such subjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C.Additionally, unless otherwise stated or the context contradicts it, the term "multitude" indicates a state in which it is plural (e.g., "a multitude of elements" indicates multiple elements). A multitude is at least two elements, but may be more if either explicitly stated or indicated by context. Furthermore, unless otherwise stated or the context otherwise makes clear, the phrase "based on" means "at least partly based on" and not "exclusively based on".

[0202] Operations of the processes described herein may be performed in any suitable order unless otherwise specified herein or the context clearly indicates otherwise. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is / are executed collectively on one or more processors by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that are executable by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-volatile computer-readable storage medium that excludes volatile signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes a non-volatile data storage circuit (e.g., buffers, cache, and queues) within transceivers of volatile signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-volatile computer-readable storage media on which executable instructions (or other memory for storing executable instructions) are stored, which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform operations described herein.In at least one embodiment, a set of non-volatile, computer-readable storage media comprises multiple non-volatile storage media. One or more of these individual non-volatile storage media lack all the code, while multiple non-volatile storage media collectively store all the code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors—for example, a non-volatile storage medium stores instructions, and a central processing unit (CPU) executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0203] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or jointly perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the performance of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0204] The use of any and all of the examples or exemplary language provided herein (e.g., "like") is intended only to better illustrate embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless otherwise claimed. No language in the specification should be interpreted as indicating that an unclaimed element is essential for the practical implementation of the disclosure.

[0205] All references, including publications, patent applications and patents cited herein, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated as incorporated by reference and set forth herein in its entirety.

[0206] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not necessarily intended as synonyms. Rather, in certain examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other but nevertheless cooperate or interact.

[0207] Unless expressly stated otherwise, terms such as "processing", "calculating", "calculating", "determining" or the like are understood to refer throughout this specification to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, such as electronic, quantities within the registers and / or memory of the computing system into other data represented likewise as physical quantities within the memory, registers or other such information storage, transmission or display devices of the computing system.

[0208] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data, which may be stored in registers and / or memory. As non-restrictive examples, "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “procedure” are used interchangeably herein, insofar as the system can embody one or more procedures and procedures can be considered as a system.

[0209] This document may refer to the receiving, acquiring, receiving, or inputting of analog or digital data into a subsystem, computer system, or computer-implemented machine. Receiving, acquiring, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as receiving data as a parameter of a function call or an application programming interface (API) call. In some implementations, the process of receiving, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data over a serial or parallel interface. In another implementation, the process of receiving, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data over a computer network from a providing entity to a receiving entity.It can also refer to the provision, output, transmission, sending, or presentation of analog or digital data. In various examples, the process of providing, outputting, transmission, sending, or presenting analog or digital data can be achieved by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0210] Although the above description presents exemplary implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. While specific distributions of responsibilities are defined above for the purpose of discussion, various functions and responsibilities could be distributed and divided in different ways depending on the circumstances.

[0211] Although the subject matter was described in a language specific to structural features and / or methodological actions, it is further understood that the subject matter claimed in the appended claims is not necessarily limited to specific features or described actions. Rather, specific features and actions are disclosed as exemplary forms of implementing the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] CN 202311036144,8

[0001]

Claims

[1] Computer-implemented method comprising: Receiving audio data corresponding to a speech utterance; Calculating, using a transformer-based audio encoder and a decoder, a weighted vector indicative of a plurality of features associated with the audio data; Calculating, using the weighted vector and one or more component vectors, an animation vector corresponding to one or more positions for one or more feature points associated with a digital character representation; and Rendering the digital character representation based at least on the animation vector. [2] The computer-implemented method of claim 1, wherein the transformer-based audio encoder is a pre-trained audio encoder, further comprising: Training the decoder based at least on the transformer-based audio encoder, locking parameters for the transformer-based audio encoder while training the decoder. [3] The computer-implemented method of claim 1, wherein the plurality of features are selected during training. [4] The computer-implemented method of claim 1, further comprising: Receiving a respective layer vector associated with the plurality of features for layers of the transformer-based audio encoder; Determining a layer weighting for individual layers; Applying the layer weighting to each individual layer; and Determine the weighted vector. [5] The computer-implemented method of claim 1, wherein the audio data has a duration less than a threshold duration. [6] The computer-implemented method of claim 5, wherein the plurality of features are determined using transformer layers. [7] The computer-implemented method of claim 1, wherein the one or more feature points correspond to at least one of facial features, a tongue position, an eye position, or a limb position. [8] The computer-implemented method of claim 1, wherein the plurality of features are extracted from the audio data by processing using a convolutional neural network (CNN). [9] The computer-implemented method of claim 1, wherein the component vector includes at least one of an emotion vector or a style vector. [10] The computer-implemented method of claim 1, wherein the decoder ignores information from one or more previous frames. [11] The computer-implemented method of claim 10, further comprising: Penalize motion between adjacent frames when the audio volume is less than a volume threshold. [12] Processor comprising: one or more processing units for: Calculating, using a transformer-based audio encoder and based at least on audio data corresponding to speech, a weighted feature vector associated with the audio data; Calculating, using the weighted feature vector and a component vector indicating one or more properties associated with the language, position data for one or more feature points of one or more deformable body components of a virtual character; and Rendering, for one or more times in a sequence of times of the audio data, image data representing the virtual character based at least on the position data to generate an animation of the character appearing to utter the speech. [13] The processor of claim 12, wherein the weighted feature vector is based at least on respective layer vectors for individual layers of the transformer-based audio encoder, wherein individual layer vectors are associated with a plurality of features extracted from the audio data. [14] The processor of claim 12, wherein parameters of the transformer-based audio encoder are locked for an associated decoder during a training process. [15] The processor of claim 12, wherein the component vector includes at least one of an emotion vector or a style vector. [16] The processor of claim 12, wherein the processor is included in at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative content operations using a large language model (LLM); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for performing one or more generative content operations using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources. [17] System comprising: one or more processing units for generating an animation of a character using position data representing one or more positions of one or more feature points of the character, the position data being calculated at least in part based on a transformer-based audio encoder processing audio data representing the speech, and component data indicating one or more values ​​corresponding to at least one of a style parameter or an emotion parameter associated with the speech. [18] The system of claim 17, wherein the transformer-based audio encoder calculates a weighted feature vector based at least on respective layer vectors for individual layers of the transformer-based audio encoder. [19] The system of claim 17, wherein parameters of the transformer-based audio encoder are locked to an associated decoder during a training process. [20] The system of claim 17, wherein the system comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative content operations using a large language model (LLM); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for performing one or more generative content operations using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources.

Images