Inferring emotions from language in audio data using deep learning

Abstract

A deep neural network can be trained to infer emotion data from input audio. The network can be a transformer-based network capable of inferring probability values ​​for a set of emotions or emotion classes. These emotion probability values ​​can be modified using one or more heuristics, such as to smooth emotion determinations over time, or via a user interface where a user can modify emotion determinations as appropriate. A user can also provide previous emotion values ​​to be combined with these emotion determination values. Specific emotion values ​​can be provided as input to an emotion-based operation, such as to provide audio-driven speech animation.

Description

BACKGROUND

[0001] There are various situations where it may be desirable to determine a type of emotion displayed by someone while uttering speech, such as speech represented by captured audio data. Certain previous approaches used machine learning to attempt to infer an emotion from input audio, however, these approaches were typically limited to the people or speakers for whom the particular model was trained and generalized poorly to other speakers. These networks were also typically based on spectrograms, requiring conversion of the audio into a spectrogram representation that would be analyzed using image-based analysis, which produced suboptimal results.Such an approach also requires multiple models to be trained for different speakers, which can be complex and computationally expensive, or may lead to varying levels of inaccuracy in the emotion inferred for any given input speech. Furthermore, previous approaches would determine a single emotion for an entire audio segment, which does not capture variations in a speaker's emotional state during that segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure will be described with reference to the drawings in which: Fig. 1 illustrates different emotions that may be displayed by a person uttering a voice line, according to at least one embodiment; Fig. 2A illustrates an example pipeline for inferring emotions from input audio, according to at least one embodiment; Fig. 2B illustrates a time window-based approach to determining emotion at specific points in an audio file or audio stream, according to at least one embodiment; Fig. 3 illustrates an example character animation system that may use emotion data inferred from input audio, according to at least one embodiment; Fig. 4A and Fig. 4B illustrate interfaces to enable specification of a previous emotion and a previous emotion strength, according to at least one embodiment; Fig. 5 illustrates an exemplary process for inferring an emotion from audio data, according to at least one embodiment; Fig. 6 illustrates components of a distributed system that can be used to infer emotions from audio, according to at least one embodiment; Fig. 7A illustrates inference and / or training logic, according to at least one embodiment; Fig. 7B illustrates inference and / or training logic, according to at least one embodiment; Fig. 8 illustrates an exemplary data center system, according to at least one embodiment; Fig. 9 illustrates a computer system, according to at least one embodiment; Fig. 10 illustrates a computer system, according to at least one embodiment; Fig. 11 illustrates at least portions of a graphics processor, according to one or more embodiments; Fig. 12 illustrates at least portions of a graphics processor, according to one or more embodiments; Fig. 13 is an example data flow diagram for an advanced compute pipeline, according to at least one embodiment; Fig. 14 is a system diagram for an example system for training, adapting, instantiating, and deploying machine learning models into an advanced compute pipeline, according to at least one embodiment; and Fig. 15A and Fig. 15B illustrates a data flow diagram for a process to train a machine learning model, as well as a client-server architecture to enhance annotation tools with pre-trained annotation models, according to at least one embodiment. DETAILED DESCRIPTION

[0003] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiment.

[0004] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), controlled and uncontrolled robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, for example and without limitation, for machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, merged reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, image processing, cloud computing, and / or other suitable applications.

[0005] Disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., an infotainment or personal digital assistance system for an autonomous or semi-autonomous machine), systems implemented using a robot, flight systems, media systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems that host one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems for performing light transport simulation,Systems for performing collaborative content creation for 3D assets, systems implemented at least in part using cloud computing resources and / or other types of systems.

[0006] Disclosed embodiments may infer emotions from speech or audio data uttered by a person or other such speaker, which may, for example, be captured in audio data using a microphone and an audio capture device that can convert a captured audio signal into digital audio data. When speaking, aspects of a person's speech may change based, at least in part, on their emotional state, similar to how a person's facial expression may change. For example, Fig. 1 Images of four example emotional states that a person might exhibit while uttering the same line of speech. This includes an image 100 showing the person in a happy state, an image 102 showing the person in an angry state, an image 104 showing the person in a disgusted state, and an image 106 showing the person in a sad state. Just as the person's facial expression changes with an emotion, there will be similar changes in the vocal expression of the speech with these changes in emotion. For various operations, it may be useful to be able to accurately and automatically identify these emotions from captured audio data.In an example use case where emotion data is to be used to generate a facial animation, the ability to accurately identify and infer a person's emotional state while uttering speech can help ensure that appropriate facial expressions, such as those shown in . Fig. 1 may be used to render the animation. Emotional state data may also be useful in other contexts, such as managing calls in a call center based, at least in part, on a detected emotional state or a change in the emotional state of at least one participant in a call. For example, an automatically generated prompt(s), script, or prompt for a call may be dynamically updated based on detected emotional states of callers.

[0007] An emotion determination system according to at least one embodiment may receive input audio data 200 as in Fig. 2A. This audio data 200 may include speech uttered by at least one person or other speaker, as captured using an audio capture device. The audio data may undergo at least some amount of preprocessing, such as to reduce background noise, remove segments of silence or non-speech, or segment the audio into audio segments, each containing speech uttered by a single speaker. This audio data may then be passed as input to an emotion determination module, apparatus, system, or process to attempt to determine or infer an emotional state of the person uttering the speech in that audio clip.In this example, the audio data 200 is passed to an algorithm that can classify the type of emotion or emotional state reflected in the uttered speech using a trained deep learning model or neural network, at least with respect to those emotional states or classes for which the model or network was trained. The neural network 202 can infer one or more emotion identifiers 204 for the input audio 200, which can then be provided as the output of the emotion determination process. This can include a single emotion identifier for individual portions of the audio data 200 or one or more emotional identifiers or determinations for an entirety of the input audio data (such as may correspond to a specific portion—such as a word or a sentence—of the received audio), among other such options.

[0008] In at least one embodiment, the neural network 202 may be a transformer-based network. This may, for example, include a network with a Wav2Vec2.0 or UniSpeech neural architecture. A transformer neural network can accurately and efficiently solve sequence-to-sequence tasks, including those with long-range dependencies. Such a network may take the audio data in an audio file format as input, rather than first having to convert the audio into an image-based representation or format—such as a spectrogram or mel-spectrogram—as required in previous approaches (which have also been found to result in less accurate results and greater instability). For example, the audio data may correspond to an audio file format, such as an uncompressed audio file format (e.g., WAV, KIFF, AU, Raw, etc.).), a lossless audio compression file format (FLAC, WavPack, TTA, ATRAC, MPEG-4, lossless WMA, SHN, etc.), or a lossy audio compression file format (e.g., Opus, MP3, Vorbis, Musepack, AAC, ATRAC, lossy WMA, etc.). Using an audio file format as opposed to converting the audio file format to a non-audio file format (e.g., an image format) enables higher accuracy and precision in predictions of the transformer-based neural network 202, while also allowing the network 202 to be more robust to different speakers. Furthermore, as described, the runtime of the system is reduced because no post-processing of the audio data into an image or other non-audio-based format is required.

[0009] In at least one embodiment, such a network may output a probability distribution, a confidence(s), and / or another output type indicating a probability of the speech represented by the input audio data corresponding to one or more of a number of emotional classes. This may include as few as two emotional classes (in which case a Boolean output may be generated by the network) or up to as many emotional classes (or combinations of classes) as can be identified and used to train the network for a given operation, application, or use case without unduly degrading performance. In at least one embodiment, such a network outputs a distribution over six emotional classes divided into (e.g.,publicly available) datasets used for training—which include anger, disgust, fear, joy, neutral emotion (or no detectable emotion), and sadness. If other datasets are used, the set of emotions may be larger or smaller, or may include a different selection of emotions based at least in part on the classifications or labels used in those datasets. It may be desirable to select a variety of datasets where possible in order to obtain a variety of examples of expressing different emotions. In at least one embodiment, a transformer-based neural network 202 will output a vector with a value for each emotion, where this value corresponds to a probability, confidence, or other indicator as to whether the speech contained in the input audio (e.g.,extracted from a video clip) was uttered by a person with a particular emotional state(s), or attempted to convey that emotion. These values ​​can be normalized to values ​​between 0 and 1 (or between 0% and 100%), all of which sum to an absolute probability value, such as 1.0 (or 100% - where the probability values, when summed, cannot exceed the maximum probability of any single emotion). For speech determined to be entirely of a single emotion, the probability vector might have values ​​of [1, 0, 0, 0, 0, 0], whereas speech determined to have equal probabilities for two different emotions might have values ​​of [0, 0.5, 0, 0.5, 0, 0].A typical output may include at least some probability for most or all emotions, such as may correspond to values ​​of [0.9; 0.02; 0.01; 0.02; 0.02; 0.03]. Other such values ​​or outputs indicating the emotional state may also be provided within the scope of the various embodiments.

[0010] In cases where more than one emotional state can be represented, each emotional state with a probability or confidence above a threshold can be identified as an emotional class represented by the language. Where it is determined that two or more emotional state classes exist, the output values ​​corresponding to each emotion can be used to weight the prominence of one emotion relative to another.

[0011] For example, if the emotional states are used to animate a virtual actor or subject, and there is a higher confidence for anger than for sadness, the virtual actor or subject can be generated to exhibit more anger than sadness, which can be reflected using different facial and / or body characteristics. Similarly, if the emotional states are used to indicate a speaker's emotions for purposes other than animating a virtual actor or subject, the strength or confidence of different emotions can be indicated to a user or system—e.g., "The speaker is more angry than sad," or "Anger: 70%; Sadness: 30%," and / or the like.

[0012] In many cases, a person's emotional state will not remain exactly the same while more than a few words of speech or other vocal sounds are uttered. Even if the emotional class might remain essentially the same, there may be periods where it is clustered with other emotions or where a particular emotion is more prominent. To account for these and other such variations, an emotion determination system may attempt to determine the emotional state at different points or times in the input audio, even potentially within the same speech uttered by a single speaker. These "points" may correspond to emotional keyframes designated for different timestamps of the audio track. An emotion detection algorithm can then output at least one emotion classification for each emotional keyframe.An operation that receives this emotional key frame data can then make decisions or perform actions based at least in part on these changes in emotion over time.

[0013] Fig. 2B illustrates an exemplary procedure for determining an emotion classification for a sequence of emotional key frames, which may be used in accordance with at least one embodiment. In this example, a fixed jump size 256 and window size 258 may be used to determine an emotional state for a sequence of frames 250, 252, 254 represented by input audio data 260. In embodiments, a jump size (or step) may also be thought of as a distance between step points determined for an audio track, where step points may be determined at a regular frequency to obtain a desired overlap and spacing of sliding windows for a single frame. In at least one embodiment, a source (e.g.,A user (a user, an application, or an operation) may specify the jump size (or stride) 256 and size of a sliding window 258 to be used to analyze the audio data. For a 16,000 Hz sample rate of input audio, an exemplary window size and stride values ​​may be set to approximately 15,000 or values ​​between 5,000 and 16,000. In this example, each audio frame may be analyzed using a number of passes through the audio data. For each pass, the audio over a given position of a window 258 may be analyzed to determine probability values ​​for a set of emotional classes. In various embodiments, a window length should be long enough to represent enough audio data to produce accurate emotional inference, but not so long as to likely encompass different emotional states, which would then lead to inaccurate ones.

[0014] Probability determinations over the entire window. For each pass, the sliding window 258 may be moved forward in time according to the specified jump size 256. The jump size 256 may be adjusted to allow at least some overlap between window positions, to have at least two windows for most points in the audio to help avoid missing emotional states, which may be relatively short. The jump size 256 may also be long enough to avoid inappropriate processing or an excessive number of passes that may be required by the audio. In at least one embodiment, a jump size may be at most half the size of the sliding window size and at least one-tenth the length of the window size to provide sufficient, but not excessive, overlaps between window positions.Similarly, there may be thresholds or ranges set for window sizes, such as at least one-tenth of a frame size or at most nine-tenths of a frame size, where a frame may be anywhere from at least about 0.1 seconds to about 10 seconds long in at least one embodiment. In at least one embodiment, probabilities for different emotional classes may be determined for each of these sliding window positions in a given frame, and then an overall probability for the frame may be determined by combining these probabilities. This may then be provided as an emotion vector for that key emotional frame, which may be considered to be positioned at a starting point, midpoint, or other location within a single frame.In this example, the frames are all the same size, so the keyframes will be relatively regular in timing with an emotion vector or classification output for each of these emotional keyframes.

[0015] In other embodiments, the frame size and keyframe location may vary based at least in part on the content of the audio. In at least one embodiment, the content of the audio may be analyzed to segment that audio according to at least one speaker, such that an audio clip contains only speech or primarily speech uttered by a single speaker (which may include editing the audio to filter out speech from other actors). In some embodiments, this may be further broken down by, for example, sentences or words included in the audio. Other factors may also be used, such as pauses in speech, changes in the volume, or changes in the speed of the uttered speech, among other such options.In at least some embodiments, thresholds may then be applied to determine window size and jump size, or an algorithm may be used to determine these sizes, as they may depend at least in part on the frame size. In some embodiments, audio (e.g., 16 kHz audio) may be analyzed until a change in an emotion is detected, or at least one change that meets or exceeds a change threshold (e.g., a confidence for a different emotion that is greater than a threshold confidence, a confidence for a different emotion that is greater than a current emotion confidence by more than a threshold, a confidence of a different emotion that is greater than the current emotion confidence by more than a threshold number of iterations, etc.) or another selection criterion is met, and then a new keyframe may be started.The previous key frame can then be analyzed using a number of passes, as previously explained, to determine an emotional vector or classification for that emotional key frame.

[0016] Emotional identifiers or classifications determined for individual emotional key frames may be provided as input to a system, service, process, application, and / or operation that performs one or more tasks based at least in part on this emotion input data. An example of such a system is an audio-driven facial animation system 300, as shown in Fig. 3. This example system may provide for automated, audio-driven animation, such as full 3D facial animation, with variable emotion control. In at least one embodiment, a collection of speech performances from one or more speech-uttering actors (e.g., specific sentences) may be captured with different emotions, levels of emotions, combinations of emotions, or styles of presentation, among other such options. Emotions supported by such a system may include any suitable emotion (or similar behavior or state) capable of being represented at least in part by character animation, image synthesis, or rendering, such as may include, but are not limited to, joy, anger, wonder, sadness, pain, or fear.A data collection process may include acquiring 4D data that includes multi-view 3D data over at least a period of speech utterance. Reconstruction of this acquired facial behavior may be performed not only for facial skin (or such surface), but also for other articulable or controllable components, elements, or features, such as the teeth, eyeballs, head, and tongue. The reconstruction may provide geometric deformation data in the temporal domain for each separately (or at least somewhat separately) modeled facial component (or other bodily component) or region. Such reconstruction may, for example, provide a full dataset for use in training a deep neural network to perform a task such as 3D facial animation.

[0017] In at least one embodiment, a trained deep neural network 306 may be based on a U-net, a generative adversarial network (GAN), or a recurrent neural network (RNN)-based architecture. Sequence-to-sequence mapping may be used to obtain a sufficiently long temporal context, which may be useful in generating a physically or behaviorally accurate animation, particularly for upper facial movement. In the exemplary system 300 of Fig. 3, a segment of audio data—such as frames or an audio segment within a current audio window 302—may be provided as input to the deep neural network 306, which may use an analysis network section 308 to analyze the audio and encode features representative of features of the audio in the audio window 302, such as may correspond to a speech segment. This analysis network section 308 may include a shared audio decoder and audio encoder for encoding audio features into an audio feature vector, which may be provided as input to an articulation network section 310 of the deep neural network 306. In this example, an emotion vector 322 (or an emotion identifier, etc.) may be provided as input.As explained in detail elsewhere herein, an emotion vector 322 may be generated using an emotion inference network 320, such as the transformer-based network 202 of FIG. Fig. 2A, which can infer an emotion from input audio for respective audio frames, windows, or segments. An emotion vector 322 may correspond to a key emotional frame for use in determining how to render one or more frames of facial animation. An emotion vector 322 may include data (e.g., probabilities, confidences, etc.) for one or more emotions that may apply to speech in audio, such as an emotion that a voice actor has been instructed to use when uttering the speech captured in the audio data. In some cases, this may include data for a single emotion identifier, such as "anger," or may include data for multiple emotions, such as "anger" and "sadness," as well as potential relative weights or probabilities for these two emotions.

[0018] In at least one embodiment, a style vector may also be provided as input to this network during training. A style vector may include data related to any aspect of the animation or facial component movement that modifies how one or more points for one or more facial components should move for a given emotion or emotion vector. This may affect the movement of specific features or facial components, or it may include providing a style of the overall animation to be used, such as "intense" or "professional." A style vector can also be thought of as providing finer-grained control over the emotion, where an emotion vector provides the identifier(s) of the emotion(s) to be used, and the style provides finer-grained control over how the emotion(s) are expressed through the animation.Other approaches to determining style data may also be used, as explained in detail elsewhere herein, for example. In different implementations, a single set of emotion and style vectors may be provided for a given audio clip, a set of vectors may be provided for each frame of an animation to be generated, or a set of vectors may be provided for specific points or frames of the animation (e.g., emotional key frames) where at least one emotion or style value or attitude is to be modified relative to a previous frame.

[0019] In this system 300, and without limitation, the emotion vector 322 is fed into a multi-level articulation section 310 of the deep neural network 306, which includes at least a beginning and an end of the network to assist in conditioning the network. The network 306 may utilize a shared audio encoder and multiple audio decoders for different facial components (e.g., facial skin, jaw, tongue, eyeballs, and head). During training, an output network section 312 of the deep neural network 306 may generate a set of vertex positions 314 and / or motion vectors (or other movements or deformations) for individual feature points of the facial components, whether for each such feature point or merely for those that have changed relative to a previous frame, among other such options.During training, these vertex positions may be compared against ground truth data, such as the original reconstructed face data from 4D image acquisition, to calculate an overall loss value. In at least one embodiment, a loss, such as an L2 loss, may be used for both the position and velocity of feature points in a representation of the output data. A loss function used to determine the loss value may include terms for position, motion, and adversarial loss in at least one embodiment. This loss value may be used during backpropagation to update network parameters for the deep neural network 306. Once it is determined that the network has converged or another training exit criterion is met (e.g.,Processing all training data or performing a target / maximum number of training iterations), the trained network 306 may be provided for inference. During inference, the network may receive only audio data 302 as input and may infer a set of vertex positions 314 for various facial components (e.g., head, face, eyeballs, jaw, tongue), which may then be fed to a renderer 316 (e.g., a rendering engine of an animation or video synthesis system) to generate an animation frame 318, which may be one of a series of frames that provide the animation upon presentation or playback. The audio data originally used by the deep neural network 306 may be the same as the original audio data generated by the transformer-based neural network 202 of FIG. Fig. 2A to determine the emotional state(s) or class(es) corresponding to the audio data. In various embodiments, the format of the audio data used by DNN 306 and transformer-based neural network 202 may be different or the same. For example, both networks 306 and 202 may use the audio without conversion to an image-based format, or network 306 may use an image-based format (e.g., a spectrogram), and network 202 may use the audio format without conversion, as described herein.

[0020] As explained in detail elsewhere herein, emotion vector data may also be provided if the generated vertex positions are to be modified in a manner related to how the deep neural network 306 would otherwise infer the vertex positions based on the audio data, for example, to convey a specific style or facial behavior to be used in inferring the vertex positions 314. In some embodiments, a deep neural network 306 may receive emotion vectors, at least when available, and use those vectors to determine how to animate a face, or use that vector in combination with its own emotion determination to attempt to provide a smoother and more accurate animation.Providing different emotional vectors for different emotional keyframes can help dynamically change the emotional expression of the rendered face over time to match the emotion conveyed in the corresponding speech data. One advantage of a transformer-based neural network 202, as described here, is that it can generalize to speech audio from many different speakers, so an operator does not need to obtain a different model trained for each speaker or type of speaker.

[0021] In at least one embodiment, changes in an emotion during a frame or audio segment may be represented in different ways. For example, if a first emotion is detected during a first half of a segment and a second emotion is detected during a second half of that segment, then two emotion vectors could be provided indicating the respective emotion during the respective time frame or for a respective emotional keyframe. In another example, a single keyframe may be generated indicating probabilities or values ​​for both emotions over that segment, such as with substantially equal probabilities. In yet other examples, the system may represent emotional values ​​for adjacent segments (e.g.,before and after) and attempt to merge or modify segments based, at least in part, on similarities or differences in emotion identification.

[0022] In some embodiments, all emotional classes may have equal (or no) weighting, so that certain probabilities can be used directly. In some embodiments, a user (or another source) may have the ability to specify at least one emotion identifier, which may then influence a weighting of at least one emotional class or influence an output emotion vector or value. For example, a user might specify that a given audio segment is associated with a "sad" emotion. During analysis, an emotion detection network might detect other emotional probabilities, such as anger or disgust.These values ​​can be used to adjust the probabilities in an emotion vector in at least one embodiment, whether by adjusting weights applied to the various probabilities to give greater weight to a "sad" emotional state, or by aggregating or averaging certain probabilities with a probability of overall sadness based on the user input, among other such options. In some embodiments, a user may also have the option of adjusting probabilities or values ​​in a given emotion vector to modify an outcome based, at least in part, on that vector.

[0023] The ability to determine an emotion from speech or voice data may also have various other applications or benefits in other contexts. For example, in a call center operation, the ability to determine the emotion of call center agents during calls may help determine whether any agents tend to display specific emotions outside of an expected or average range, which can help identify agents who may benefit from further training or assistance.An ability to detect a strong angry or sad emotion could also trigger a request for that agent to take a break or perform a different task, or could cause other calls to be rerouted to that agent that may help improve the agent's emotional state or that might be more consistent with that emotional state. Emotional state data for a call can also be logged, so that if a customer has a complaint about an agent being angry or rude on the call, the emotional state data can be analyzed to determine if the complaint may be legitimate.

[0024] Such data can also be useful in analyzing the speech of a customer or someone outside the call center. For example, if it can be determined that a caller is becoming angry during a call, the call center could decide to reroute that call to another agent, such as a manager or someone better trained to handle specific emotions or emotional states. Similarly, data stored for a call can help verify a caller's emotional state during the call, which could assist with tasks such as verifying information about a complaint or help train agents based at least in part on a caller's emotional state during a call.If the emotional state can be determined through an initial menu of options that the customer navigates through using voice commands, then that call can be initially rerouted based on the caller's emotional state, or can provide a call center agent with advance information about the emotional state, which can help the agent better prepare for and manage the call. For call centers where agents read at least some of their responses from a script, the emotional state could help select a script that is more accurate for the current situation, such as using softer language if it is inferred that a customer is angry, or more supportive language if the customer is determined to be sad, and so on.A caller's emotional state can also be useful where the call center uses virtual bots or assistants—at least initially—to determine where to redirect calls. Instead of proceeding with a fully automated call, for example, the call can be transferred to a live agent if the caller is determined to be angry, frustrated, or the like.

[0025] In at least some embodiments, an ability to change an emotional state with individual keyframes, as well as an ability to adjust the locations or frequency of those keyframes, can result in emotional changes that may not seem natural when displayed. For example, a speaker might start a long sentence that is more sad than angry, but then transition to being more angry than sad. There may also be a determination mid-sentence that for a given keyframe, the speaker is exhibiting a different emotion than for the rest of the sentence. However, rapid emotional changes may involve choppy transitions or at least not consistent with actual human behavior, where emotional transitions may be at least somewhat gradual.Approaches in at least some embodiments may use one or more of a number of heuristics or post-processing operations to attempt to smooth out inaccurate predictions of a model, as well as to provide more natural transitions between emotional states (e.g., a person rarely goes from 100% sad immediately to 100% angry in the middle of a sentence). In at least one embodiment, this may include using a sliding window approach, such as the approach explained with respect to audio data, except for using the sliding windows with respect to key frames specific to the audio. This may include performing smoothing for at least non-neutral emotions over a number of key frames, where this number (e.g.,2-10) of keyframes may be specified by a user or application, or may be determined based at least in part on a number or frequency of keyframes determined for an audio clip, among other such options.

[0026] In one embodiment, a system may allow a user, application, or other such source to specify or set an emotion strength value. For example, a user may select a ratio from 0 to 1 representing an emotion strength. In at least one embodiment, a greater emotion strength value corresponds to a higher level of expressiveness of the corresponding emotion. If the strength is set to 0, this may indicate that no expressiveness of that emotion is to be used. For example, in a video game, it may be desired that an evil character not display sadness or happiness, and a user may specify an emotion strength value of 0 for those emotions, so that the character is only intended to express things with, for example, an angry, disgusted, or neutral emotion.If a character is intended to be very happy, a user could set an emotion strength for a happy emotion close to 1 (e.g., 0.9) and much lower values ​​for other emotion strengths. Such approaches can not only smooth out emotion determinations but can also provide emotion determinations more appropriate for a given character.

[0027] Available heuristics may also allow for specification or setting related to previous emotions. In this context, a "previous" emotion does not refer to a previously determined or demonstrated emotion in an audio file, but instead refers to an emotion or emotional state determined prior to dynamic analysis, for example, by a Transformer-based neural network 202. This may include an emotion specified for a given speech sample in audio data, for example, by a user, application, or operation. For example, where the emotion determinations are used to generate a facial animation, a user might specify that the character animated during that speech should appear sad. However, as mentioned, using only a single emotion across an entire speech sample may not appear natural.A system may then allow a user (or other such source), for example, to specify a previous emotion to use in a speech sample, but also inferring changes in the emotion for different key frames during that speech. The emotion and the current emotion values ​​can then be merged so that the character will demonstrate the previous emotion, but this emotion can be merged with other emotions at different times during the speech, such as appearing more or less sad at different times, being merged with a neutral value or somewhat angry over a section of speech, and so on. To allow some control over this merge, a previous emotion strength value can also be supplied.This can act as a type of weight to indicate how much of that previous emotion value should be merged with an emotion determined by a neural network, where a previous emotion strength of 0.9 might cause the emotion value to primarily reflect the previous emotion, while a previous emotion strength of around 0.3 might cause the emotion value to reflect at least some of the previous emotion at all times during speech, which may at least consistently provide some smoothing of the emotion determinations of the speech.

[0028] Fig. 4A and Fig. 4B illustrate example states 400, 450 of a user interface that can be used to specify emotions for training data, as well as to provide style or modification data for emotion determination at inference time, among other such options. Upon specifying or modifying emotion data, an animation, rendering, or reconstruction representative of one or more particular emotion probability values ​​406 may be displayed. A user viewing this interface can then make any value adjustments determined to be appropriate. For example, an emotion determination may primarily be angry, but a listener may also interpret the speech utterance as sounding somewhat sad. To more accurately label the data, a user can adjust the label applied so that the network more accurately learns to interpret emotions in audio data.As illustrated, a time point 404 (e.g., a location of a key frame) in the audio data 402 may be specified for which these settings are to be applied. As mentioned, a single setting could be used for an audio clip or segment, however, in other situations the emotional state may change during such a clip or segment, such as at different times or for / at specific animation frames, which may be referred to as emotional key frames. An emotional key frame may indicate when one or more values ​​for an emotion or style should change, and corresponding input vectors with these values ​​may be provided as input to a network during training to learn these changes.

[0029] A user of this interface may also specify a previous emotion 408 to be merged with the emotional determination. A user may also specify a previous emotion strength 410, which may be used to determine a merge weight for this previous emotion with respect to a particular emotion. As in Fig. 4A, the previous emotion value of "angry" has a corresponding previous emotion strength value of 0.0. Accordingly, the emotional state illustrated in the rendered image 412 is primarily joy, as determined by the particular emotion settings 406 or probabilities. As shown in Fig. 4B, a user setting the previous emotion strength value 452 to 0.6 results in an anger emotion being merged with the joy (and neutral) emotion determinations, resulting in an emotional state, as illustrated in the rendered image 454, which is an equal merger of joy and anger, such as when the user is happy with a result but annoyed with the procedure used to obtain that result. As mentioned, an emotion strength for each individual emotion can also be provided and used to smooth emotions or modify emotional determinations, among other such options. An interface, such as the one in Fig. 4A may also allow a user to set values ​​for emotions and / or previous emotions and related values ​​at different keyframes or points 404 in the audio. Such an interface may also allow a user to select which heuristics to apply to a given audio clip, as well as any values ​​that may be used to modify or control the manner in which these heuristics are applied.

[0030] As mentioned, such an interface can be used at inference time as a type of post-process, and can also be used for continued learning in at least some embodiments. For example, a user can view the generated animation playback through this interface, where an animation of the character is present. Fig. 4B, if the user thinks the animation contains too much intensity for the situation, the user can adjust the intensity style selector to decrease the intensity and re-render the frame(s) of the animation. If the user detects a hint of sadness in the character's speech that is not captured in the animation, the user can adjust that setting as well. In some embodiments, a user may also be able to provide, as a type of style input, a setting to specific feature points or facial components in the display. For example, the user may use a pointer to grab and move a position of the character's lip, and this information can be used as style input for re-rendering the animation.Other changes may also be provided, such as head movement, head tilt, eye movement, or eye focus, or other such changes that may be conveyed through emotion or style input for re-rendering (or updated rendering, or updated synthesis) of the animation. Various other animation control parameters may also be specified through such an interface, which may affect the final rendering.

[0031] In some embodiments, the transformer-based neural network 202 and the deep neural network 306 can be trained in an end-to-end manner, where outputs from the deep neural network 306 can be used to update parameters not only of the network 306, but also of the network 202. For example, where the probability or confidence for a particular segment of audio data is determined to be very high (e.g., 0.9) for anger, but the animated character animated using 0.9 for anger as an input to the network 306 is too expressive or less human-like, this feedback can be used to adjust the parameters of the network 202 to train the network 202 to instead predict lower confidences for anger (e.g., 0.7) or probabilities for similar speech types.In this way, the emotional states or classes (and their corresponding confidences) can be fine-tuned to help the network 306 generate animations that more closely or precisely resemble emotions.

[0032] Fig. 5 illustrates an exemplary process 500 for inferring emotions from an input audio clip, which may be used in accordance with at least one embodiment. It should be understood that for this and other processes presented herein, there may be additional, fewer, or alternative steps performed in similar or alternative orders or at least partially in parallel within the scope of the various embodiments, unless specifically stated otherwise. In this process, audio data is obtained 502 representing speech uttered by at least one speaker, such as at least one human uttering speech during a conversation. This speech may have been captured by an audio capture device, such as at least one microphone or microphone array, and then converted into digital audio data.This audio data may be divided 504 into speech segments (e.g., sentences, paragraphs, words, or utterances between pauses), each uttered or labeled as corresponding to a single speaker (e.g., where there are multiple speakers but one speaker is prominent). An audio segment may be selected 506 for emotion analysis and provided 508 as input to a Transformer-based neural network or other such emotion determination network or algorithm. One or more frames of the segment may be analyzed 510 using the neural network to infer probability values ​​(or other values) for a set of emotions. This may include a fixed set of emotions for which the neural network has been trained, as well as potentially additional emotions that the network has learned through continued learning, among other such options.The number of frames in the segment may depend on a number of factors, such as the length or content of the segment, as well as the window or step size for analysis. For each frame of the segment, an emotion vector may be received 512 indicating probabilities for the set of emotions or at least a subset of the emotions. A determination may be made 514 as to whether any heuristics should be applied to the emotion vector(s). If so, one or more of these heuristics may be applied 516 to the vectors to perform smoothing or emotion determination adjustments, among other such options. In some embodiments, this may include adjusting the emotion probability values ​​based on a previous emotion and / or emotion strength, as explained herein.The emotion vectors may be provided 518, following some heuristic, to an application (or other recipient or target) for use in performing one or more emotion-based tasks or analyses. A determination may be made 520 as to whether there are further segments to be analyzed, and if so, this process may proceed to a next segment. In some embodiments, segment analysis may be performed for at least some of the segments in parallel. After emotion vectors are provided, a user may be permitted 522 to review and modify values ​​in these emotion vectors as appropriate, such as to implement any settings deemed appropriate by the user for a given emotion-based task.

[0033] As an example, emotional vectors can be annotated with a facial animation process that attempts to produce an animation with realistic behavior for different emotional states for a variety of different character types. This may include, for example, an audio-driven full three-dimensional (3D) facial animation with emotion control. Using such an approach, a realistic animation can be produced without requiring any manual input or post-processing—although possible where desired. Automating such animations can help significantly reduce the time, skill, and cost required for manual (or at least partially manual) character animation.Audio-triggered facial animation can provide an efficient way to create facial animation compared to traditional approaches, since only audio data is needed to trigger the animation of a given character.

[0034] Various systems can also support realignment. In realignment, the movement of one character can be mapped to the movement of another character, so that a similar animation can be created for similar emotions and / or a similar style. An interface, such as in Fig. 4A and Fig. 4B may also be useful in a remapping context, where different characters might express emotions or styles in slightly different ways. A user may be able to load different characters into this interface and view how a remapping would appear for that character, and may then modify one or more aspects, movement styles, or behaviors for that specific character or character type.

[0035] As explained, aspects of various approaches presented herein may be lightweight enough to be executed in real-time or near-real-time on a device such as a client device, e.g., a personal computer or a game console. Such processing may be performed on content (e.g., a rendered version of a unique asset) generated on or received from that client device or an external source, such as streaming sensor data or other content received over at least one network. In some cases, the processing and / or determination of that content may be performed by one of these other devices, systems, or entities, which may then be provided to the client device (or other such recipient) for presentation or other such use.

[0036] As an example, Fig. 6 illustrates an example network configuration 600 that may be used to provide, create, modify, encode, and / or transmit data or other such content. In at least one embodiment, a client device 602 may create or receive data for a session using components of a content application 604 executing on the client device 602 and data stored locally on that client device. In at least one embodiment, a content application 624 executing on a server 620 (e.g., a cloud server or edge server) may initiate a session associated with at least the client device 602, which may utilize a session manager and user data stored in a user database 634, and may cause content 632 to be determined by a content manager 626.A content manager 626 may operate with an audio-to-face module 628 or system to determine facial animation corresponding to the input audio, as well as an emotion application 630 that may perform one or more tasks using certain emotion data. This may include, for example, using the audio data to generate image, video, or other visual representation data using an asset (e.g., a character mesh) from an asset database 632 to an extent permitted by a rights manager 630 or other such component or service. At least a portion of this generated content (separate and distinct from the assets themselves) may be transmitted to the client device 602 using a suitable transmission manager 622 for delivery via download, streaming, or other such transmission channel.An encoder may be used to encode or compress at least some of this data prior to transmission to the client device 602. In at least one embodiment, the client device 602 receiving such content may provide this content to a corresponding content application 604, which may also or alternatively include a graphical user interface (GUI) 610, an audio-to-face component 612, and an emotion application 614 for use in performing emotion-based tasks. A decoder may also be used to decode data received over the network(s) 640 for presentation via the client device 602, such as image or video content through a display 606, and audio, such as sounds and music, through at least one audio playback device 608, such as speakers or headphones.In at least one embodiment, at least a portion of this content may already be stored, rendered on, or accessible to the client device 602, such that transfer over the network 640 is not required for at least this portion of the content, such as where this content was previously downloaded or stored locally on a hard drive or optical disk. In at least one embodiment, a transfer mechanism, such as data streaming, may be used to transfer this content from the server 620 or user database 634 to the client device 602. In at least one embodiment, at least a portion of this content may be streamed from another source, such as a third-party service provider 660 or another client device 650, which may also include a content application 662 for creating or delivering content.In at least one embodiment, portions of this functionality may be performed using multiple computing devices or multiple processors within one or more computing devices, which may include, for example, a combination of CPUs and GPUs.

[0037] In this example, these client devices may include any suitable computing device, such as a desktop computer, a notebook computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR / AR / MR headset, VR / AR / MR glasses, a wearable computer, or a smart TV. Each client device may submit a request over at least one wired or wireless network, such as the Internet, an Ethernet, a local area network (LAN), or a cellular network, among other such options. In this example, these requests may be submitted to an address associated with a cloud provider that may operate or control one or more electronic resources in a cloud provider's environment, such as a data center or server farm.In at least one embodiment, the request may be received or processed by at least one edge server located at a network edge and outside of at least one security layer associated with the cloud provider's environment. In this way, latency may be reduced by allowing client devices to interact with servers located in closer proximity, while also improving the security of resources in the cloud provider's environment.

[0038] In at least one embodiment, such a system may be used to perform graphical rendering operations. In other embodiments, such a system may be used for other purposes, such as providing image or video content to test or validate autonomous machine applications, or performing deep learning operations. In at least one embodiment, such a system may be implemented using an edge device or may integrate one or more virtual machines (VMs). In at least one embodiment, such a system may be implemented at least partially in a data center or at least partially using cloud computing resources. INFERENCE AND TRAINING LOGIC

[0039] Fig. 7A illustrates inference and / or training logic 715 used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are described below in connection with Fig. 7A and / or Fig. 7B provided.

[0040] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 701 to store feedforward and / or output weighting and / or input / output data and / or other parameters to configure neurons or layers of a neural network being trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be wired to the code and / or data storage 701 to store graph code or other software to control the timing and / or order in which weighting and / or other parameter information is to be loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, the code, such as graph code, loads weighting or other parameter information based on a neural network architecture to which the code corresponds to, onto the processor ALUs. In at least one embodiment, the code and / or data storage 701 stores the weighting parameters and / or input / output data of each layer of a neural network trained or used in connection with one or more embodiments during forward propagation of input / output data and / or weighting parameters during training and / or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included in other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0041] In at least one embodiment, any portion of code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 701 may be cache memory, dynamic randomly addressable memory (DRAM), static randomly addressable memory (SRAM), non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the choice of whether the code and / or code and / or data storage 701 is, for example, internal or external to a processor or consists of DRAM, SRAM, flash memory, or another type of storage may depend on the available on-chip storage versus off-chip storage, latency requirements of training and / or inference functions performed, the batch size of data used in inferring and / or training a neural network, or a combination of these factors.

[0042] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 705 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data storage 705 stores weighting parameters and / or input / output data of each layer of a neural network trained in connection with one or more embodiments during backpropagation of input / output data and / or weighting parameters used during training and / or inference using aspects of one or more embodiments.

[0043] In at least one embodiment, the training logic 715 may include or be wired to a code and / or data storage 705 to store graph code or other software to control the timing and / or order in which weight and / or other parameter information is to be loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as the graph code, loads weight or other parameter information to the processor ALUs based on a neural network architecture corresponding to the code. In at least one embodiment, any portion of the code and / or data storage 705 may be included with other on-chip or off-chip data storage, including in an L1, L2, or L3 cache or system memory of a processor.In at least one embodiment, any portion of code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the choice of whether the code and / or data storage 705 is, for example, internal or external to a processor, or consists of DRAM, SRAM, flash memory, or other storage type, may depend on the available on-chip versus off-chip storage, latency requirements of training and / or inference functions performed, the batch size of data used in inferring and / or training a neural network, or a combination of these factors.

[0044] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 caches or system memory.

[0045] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, one or more arithmetic logic units (ALU(s)) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations based at least in part on or specified by training and / or inference code (e.g., graph code), a result of which may generate activations (e.g., output values ​​of layers or neurons within a neural network) stored in the activation storage 720 that are functions of input / output and / or weighting parameter data stored in the code and / or data storage 701 and / or the code and / or data storage 705.In at least one embodiment, activations stored in activation storage 720 are generated according to linear algebraic and / or matrix-based mathematics performed by ALU(s) 710 in response to the execution of instructions or other code, using weight values ​​stored in code and / or data storage 705 and / or code and / or data storage 701 as operands along with other values, such as bias values, gradient information, moment values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705 or code and / or data storage 701 or other storage on-chip or off-chip.

[0046] In at least one embodiment, the ALU(s) 710 is / are included in one or more processors or other hardware logic devices or circuits, whereas in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses it (e.g., a coprocessor). In at least one embodiment, the ALUs may be included within execution units of a processor or otherwise in a bank of ALUs that can be accessed by the execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.).In at least one embodiment, the code and / or data storage 701, the code and / or data storage 705, and the enable storage 720 may be located on the same processor or other hardware logic device or circuit, whereas in another embodiment, they may be located on different processors or other hardware logic devices or circuits, or in a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the enable storage 720 may be included in other on-chip or off-chip data storage, including in an L1, L2, or L3 cache or system memory of a processor.Furthermore, the inference and / or training code may be stored with other code accessible to a processor or other hardware logic or circuitry, and may be retrieved and / or processed using fetch, decode, scheduling, execution, elimination, and / or other logic circuitry of a processor.

[0047] In at least one embodiment, the activation storage 720 may be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the activation storage 720 may be internal or external, in whole or in part, to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation storage 720 is, for example, internal or external to a processor, or consists of a DRAM, an SRAM, flash memory, or another type of storage, may depend on the available on-chip storage versus off-chip storage, the latency requirements of the training and / or inference functions performed, the batch size of the data used in inferencing and / or training a neural network, or a combination of these factors.In at least one embodiment, the method described in . Fig. 7A may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or a Nervana® processor (e.g., “Lake Crest”) from Intel Corp. In at least one embodiment, the inference and / or training logic 715 illustrated in Fig. 7A may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as field programmable gate arrays (“FPGAs”).

[0048] Fig. 7B illustrates inference and / or training logic 715 according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the inference and / or training logic 715 may Fig. 7B may be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's Tensorflow® processing unit, a Graphcore™ inference processing unit (IPU), or a Nervana® processor (e.g., "Lake Crest") from Intel Corp. In at least one embodiment, the inference and / or training logic 715 illustrated in Fig. 7B may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, code and / or data storage 701 and code and / or data storage 705, which may be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, moment values, and / or other parameter or hyperparameter information. In at least one embodiment, Fig. 7B, each of the code and / or data storage 701 and the code and / or data storage 705 is associated with a dedicated compute resource, such as the compute resource 702 and the compute resource 706, respectively. In at least one embodiment, each of the compute resource 702 and the compute resource 706 includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data storage 701 and the code and / or data storage 705, respectively, the result of which is stored in the enable storage 720.

[0049] In at least one embodiment, each of the code and / or data storage 701 and 705 and the corresponding computational resource 702 and 706 corresponds to different layers of a neural network, such that the resulting activation of a "storage / compute pair 701 / 702" of the code and / or data storage 701 and the computational resource 702 is provided as input to the "storage / compute pair 705 / 706" of the code and / or data storage 705 and the computational resource 706 to mirror the conceptual organization of a neural network. In at least one embodiment, each of the storage / compute pairs 701 / 702 and 705 / 706 may correspond to more than one layer of a neural network. In at least one embodiment, additional storage / compute pairs (not shown) may be included after or in parallel with storage / compute pairs 701 / 702 and 705 / 706 in inference and / or training logic 715. DATA CENTER

[0050] Fig. Figure 8 illustrates an example data center 800 in which at least one embodiment may be used. In at least one embodiment, data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0051] In at least one embodiment, as in Fig. 8, the data center infrastructure layer 810 may include a resource orchestrator 812, clustered compute resources 814, and node compute resources ("node RRs") 816(1)-816(N), where "N" represents any positive integer. In at least one embodiment, the node RRs 816(1)-816(N) may include any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and cooling modules, etc. In at least one embodiment, one or more of the node RRs 816(1)-816(N) may be a server having one or more of the computing resources mentioned above.

[0052] In at least one embodiment, the grouped computing resources 814 may include separate groupings of node RRs housed in one or more frames (not shown), or multiple frames housed in data centers at various geographical locations (also not shown). Separate groupings of node RRs within grouped computing resources 814 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs, including CPUs or processors, may be grouped within one or more frames to provide computing resources to support one or more workloads.In at least one embodiment, one or more frames may also include any number of power modules, cooling modules, and network switches in any combination.

[0053] In at least one embodiment, resource orchestrator 812 may configure or otherwise control one or more node RRs 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure ("SDI") management entity for data center 800. In at least one embodiment, the resource orchestrator may include hardware, software, or some combination thereof.

[0054] In at least one embodiment, as in Fig. 8, the framework layer 820 includes a task scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 may include a framework to support software 832 of the software layer 830 and / or one or more applications 842 of the application layer 840. In at least one embodiment, the software 832 or the application(s) 842 may each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 may be, but is not limited to, some type of free and open source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which may utilize the distributed file system 828 for large-scale data processing (e.g., "Big Data").In at least one embodiment, the task scheduler 822 may include a Spark driver to facilitate scheduling workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 may be capable of configuring various layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 828, to support large-scale data processing. In at least one embodiment, the resource manager 826 may be capable of managing clustered or grouped compute resources allocated or allocated to support the distributed file system 828 and the task scheduler 822. In at least one embodiment, the clustered or grouped compute resources may include a clustered compute resource 814 on the data center infrastructure layer 810.In at least one embodiment, the resource manager 826 may coordinate with the resource coordinator 812 to manage these allocated or assigned computing resources.

[0055] In at least one embodiment, the software 832 included in software layer 830 may include software used by at least portions of node RRs 816(1) through 816(N), clustered computing resources 814, and / or distributed file system 828 of framework layer 820. The one or more types of software may include, but are not limited to, Internet website search software, email virus scanning software, database software, and video content streaming software.

[0056] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by at least portions of node RRs 816(1) through 816(N), clustered computing resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive computing application, and a machine learning application that includes training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in connection with one or more embodiments.

[0057] In at least one embodiment, any one of the configuration manager 824, the resource manager 826, and the resource coordinator 812 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. In at least one embodiment, the self-modifying actions may relieve a data center operator of the data center 800 from potentially making poor configuration decisions and potentially avoid underutilized and / or malfunctioning portions of a data center.

[0058] In at least one embodiment, data center 800 may include tools, services, software, or other resources to train one or more machine learning models, or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computing resources described above with respect to data center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 800 using weighting parameters calculated using one or more training techniques described herein.

[0059] In at least one embodiment, the data center may use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0060] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are described below in connection with Fig. 7A and / or Fig. 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 8 for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0061] Such components can be used to determine one or more emotion values ​​from audio data. COMPUTER SYSTEMS

[0062] Fig. 9 is a block diagram illustrating an example computer system, which may be a system 900 with interconnected devices and components, a system-on-a-chip (SOC), or some combination thereof, and formed with a processor that may include execution units to execute an instruction, according to at least one embodiment. In at least one embodiment, the computer system 900 may include, without limitation, a component, such as a processor 902, for employing execution units that include logic to perform algorithms on process data, according to the present disclosure, such as in the embodiment described herein.In at least one embodiment, computer system 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ available from Intel Corporation of Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, and the like) may be used. In at least one embodiment, computer system 900 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0063] The embodiments may be used on other devices, such as handheld devices and embedded applications. Some examples of portable devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a digital signal processor ("DSP"), a system on a chip, network computers ("NetPCs"), set-top boxes, network hubs, wide area network ("WAN") switches, or any other system capable of performing one or more instructions, according to at least one embodiment.

[0064] In at least one embodiment, computer system 900 may include, without limitation, a processor 902, which may include, without limitation, one or more execution units 908 to perform training and / or inference of a machine learning model according to the techniques described herein. In at least one embodiment, computer system 900 is a single-processor desktop or server system, however, in another embodiment, computer system 900 may be a multi-processor system. In at least one embodiment, processor 902 may include, without limitation, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as, for example, a digital signal processor.In at least one embodiment, the processor 902 may be coupled to a processor bus 910 that may transmit data signals between the processor 902 and other components in the computer system 900.

[0065] In at least one embodiment, processor 902 may include, without limitation, an internal Level 1 ("L1") cache ("cache") 904. In at least one embodiment, processor 902 may include a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache may be external to processor 902. Other embodiments may also include a combination of both internal and external caches depending on the particular implementation and requirements. In at least one embodiment, register file 906 may store various types of data in various registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0066] In at least one embodiment, execution unit 908, which includes, without limitation, logic to perform integer and floating-point operations, is also located in processor 902. In at least one embodiment, processor 902 may also include microcode ("uCode") read-only memory (ROM) that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 908 may include logic to handle a compressed instruction set 909. In at least one embodiment, by incorporating a compressed instruction set 909 into an instruction set of a general-purpose processor 902, along with associated circuitry to execute instructions, operations used by numerous multimedia applications may be performed using compressed data in a general-purpose processor 902.In one or more embodiments, numerous multimedia applications may be accelerated and executed more efficiently by using the full width of a processor data bus to perform operations on compressed data, which may eliminate the need to transfer smaller units of data across the processor data bus to perform one or more operations on only one piece of data at a time.

[0067] In at least one embodiment, execution unit 908 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other logic circuits. In at least one embodiment, computer system 900 may include, without limitation, a memory 920. In at least one embodiment, memory 920 may be implemented as a dynamic random access memory (DRAM) device, a static random access memory (“SRAM”) device, a flash memory device, or another storage device. In at least one embodiment, memory 920 may store instruction(s) 919 and / or data 921 represented by data signals that may be executed by processor 902.

[0068] In at least one embodiment, a system logic chip may be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub (“MCH”) 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high-bandwidth memory path 918 to the memory 920 for storing instructions and data and for storing graphics commands, data, and textures. In at least one embodiment, the MCH 916 may route data signals between the processor 902, the memory 920, and other components in the computer system 900, and may bridge data signals between the processor bus 910, the memory 920, and a system I / O 922.In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 916 may be coupled to memory 920 via a high-bandwidth memory path 918, and a graphics / video card 912 may be coupled to the MCH 916 via an Accelerated Graphics Port ("AGP") interconnect 914.

[0069] In at least one embodiment, computer system 900 may use system I / O 922, which is a proprietary hub interface bus, to couple MCH 916 to I / O controller hub ("ICH") 930. In at least one embodiment, ICH 930 may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripherals to memory 920, chipset, and processor 902. Examples may include, without limitation, an audio controller 929, a firmware hub (“flash BIOS”) 928, a wireless transceiver 926, a data storage device 924, a legacy I / O controller 923 including user input and keyboard interfaces 925, a serial expansion port 927, such as a universal serial bus (“USB”), and a network controller 934.The data storage device 924 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0070] In at least one embodiment, Fig. 9 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 9 may illustrate an exemplary system on a chip ("SoC"). In at least one embodiment, the devices may be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 900 are interconnected using Compute Express Link (CXL) interconnects.

[0071] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are described below in connection with Fig. 7A and / or Fig. 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 9 for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0072] Such components can be used to determine one or more emotion values ​​from audio data.

[0073] Fig. 10 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0074] In at least one embodiment, system 1000 may include, without limitation, processor 1010 communicatively coupled to any number or type of components, peripherals, modules, or devices. In at least one embodiment, processor 1010 is coupled using a bus or interface, such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ("SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, 3), or a Universal Asynchronous Receiver / Transmitter ("UART") bus. In at least one embodiment, Fig. 10 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 10 may illustrate an exemplary system on a chip (“SoC”). In at least one embodiment, the Fig. 10 may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of Fig. 10 interconnected using Compute Express Link (CXL) interconnects.

[0075] In at least one embodiment, Fig. 10 a display 1024, a touch screen 1025, a touchpad 1030, a near field communications unit (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, an express chipset (“EC”) 1035, a trusted platform module (“TPM”) 1038, a BIOS / firmware / flash memory (“BIOS, FW-Flash”) 1022, a DSP 1060, a drive 1020, such as a solid state drive (“SSD”) or a hard disk drive (“HDD”), a wireless local area network unit (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide area network unit (“WWAN”) 1056, a global positioning system (GPS) 1055, a camera (“USB 3.0 camera”) 1054, such as a USB 3.0 camera and / or a low-power double data rate ("LPDDR") memory unit ("LPDDR3") 1015, implemented, for example, in an LPDDR3 standard. These components may each be implemented in a suitable manner.

[0076] In at least one embodiment, other components may be communicatively coupled to processor 1010 via previously discussed components. In at least one embodiment, an accelerometer 1041, an ambient light sensor ("ALS") 1042, a compass 1043, and a gyroscope 1044 may be communicatively coupled to sensor hub 1040. In at least one embodiment, a thermal sensor 1039, a fan 1037, a keyboard 1046, and a touchpad 1030 may be communicatively coupled to EC 1035. In at least one embodiment, a speaker 1063, a headset 1064, and a microphone ("mic") 1065 may be communicatively coupled to an audio unit ("Class-D audio codec and amplifier") 1062, which in turn may be communicatively coupled to DSP 1060. In at least one embodiment, the audio unit 1064 may include, for example and without limitation, an audio encoder / decoder ("codec") and a Class D amplifier.In at least one embodiment, a SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, may be implemented in a next generation form factor (NGFF).

[0077] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are described below in connection with Fig. 7A and / or Fig. 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 10 for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0078] Such components can be used to determine one or more emotion values ​​from audio data.

[0079] Fig. 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform embodied in a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0080] In at least one embodiment, system 1100 may include or be incorporated into a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1100 may also include, be coupled to, or integrated with a wearable device, such as a wearable smart watch device, a smart eyeware device, an augmented reality device, or a virtual reality device.In at least one embodiment, processing system 1100 is a television or set-top box device that includes one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0081] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 to process instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of one or more processor cores 1107 is configured to execute a specific instruction set 1109. In at least one embodiment, the instruction set 1109 may enable Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Word (VLIW) computing. In at least one embodiment, the processor cores 1107 may each process a different instruction set 1109, which may include instructions to enable emulation of other instruction sets.In at least one embodiment, the processor core 1107 may also include other processing devices, such as a digital signal processor (DSP).

[0082] In at least one embodiment, processor 1102 includes a cache memory 1104. In at least one embodiment, processor 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared by various components of processor 1102. In at least one embodiment, processor 1102 also uses an external cache (e.g., a Level 3 (L3) cache or Last Level Cache (LLC)) (not shown), which may be shared by processor cores 1107 using known cache coherence techniques. In at least one embodiment, register file 1106 is additionally included in processor 1102, which may include various types of registers for storing various types of data (e.g., integer registers, floating-point registers, status registers, and an instruction pointer register).In at least one embodiment, register file 1106 may include general-purpose registers or other registers.

[0083] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 for communicating communication signals, such as address, data, or control signals, between the processor 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1110 is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other interface buses. In at least one embodiment, the processor(s) 1102 include an integrated memory controller 1116 and a platform controller hub 1130.In at least one embodiment, the storage controller 1116 enables communication between a storage device and other components of the system 1100, while the platform controller hub (PCH) 1130 provides connections to I / O devices via a local I / O bus.

[0084] In at least one embodiment, memory device 1120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device having suitable performance to serve as process memory. In at least one embodiment, memory device 1120 may operate as system memory for system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 execute an application or process. In at least one embodiment, memory controller 1116 is also coupled to an optional external graphics processor 1112 that can communicate with one or more graphics processors 1108 in processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 may be connected to the processor(s) 1102. In at least one embodiment, the display device 1111 may comprise one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 may comprise a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications.

[0085] In at least one embodiment, the platform controller hub 1130 enables peripherals to connect to the storage device 1120 and the processor 1102 via a fast I / O bus. In at least one embodiment, the I / O peripherals include, without limitation, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors 1125 may include touch screen sensors, pressure sensors, or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 may be a WiFi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 may allow a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1110. In at least one embodiment, the audio controller 1146 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling legacy devices (e.g.,Personal System 2 (PS / 2) devices) to the system. In at least one embodiment, the platform controller hub 1130 may also be connected to one or more universal serial bus (USB) controllers 1142, interconnect input devices such as keyboard and mouse 1143 combinations, a camera 1144, or other USB input devices.

[0086] In at least one embodiment, an instance of the memory controller 1116 and the platform controller hub 1130 may be integrated into a discrete external graphics processor, such as an external graphics processor 1112. In at least one embodiment, the platform controller hub 1130 and / or memory controller 1116 may be external to the one or more processors 1102. For example, in at least one embodiment, the system 1100 may include an external memory controller 1116 and a platform controller hub 1130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with the one or more processors 1102.

[0087] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are described below in connection with Fig. 7A and / or Fig. 7B. In at least one embodiment, portions of or all of the inference and / or training logic 715 may be integrated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may utilize one or more of the ALUs embodied in a graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that described in Fig. 7A or Fig. 7B. In at least one embodiment, weighting parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of a graphics processor to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0088] Such components can be used to determine one or more emotion values ​​from audio data.

[0089] Fig. 12 is a block diagram of a processor 1200 including one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores, up to and including the additional core 1202N, represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0090] In at least one embodiment, internal cache units 1204A-1204N and shared cache units 1206 represent a cache hierarchy within processor 1200. In at least one embodiment, cache units 1204A-1204N may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, with a highest cache level prior to external memory classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 1206 and 1204A-1204N.

[0091] In at least one embodiment, the processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functionality for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external memory devices (not shown).

[0092] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for a concurrent multithreaded element. In at least one embodiment, the system agent core 1210 includes components for coordinating and operating the cores 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU) that includes logic and components to regulate one or more power states of the processor cores 1202A-1202N and the graphics processor 1208.

[0093] In at least one embodiment, processor 1200 additionally includes a graphics processor 1208 to perform graphics processing operations. In at least one embodiment, graphics processor 1208 is coupled to shared cache units 1206 and system agent core 1210, which includes one or more integrated memory controllers 1214. In at least one embodiment, system agent core 1210 also includes a display controller 1211 to drive the graphics processor output to one or more coupled displays. In at least one embodiment, display controller 1211 may also be a separate module coupled to graphics processor 1208 via at least one interconnect, or may be integrated within graphics processor 1208.

[0094] In at least one embodiment, a ring-based interconnect 1212 is used to couple internal components of processor 1200. In at least one embodiment, an alternative interconnect, such as a point-to-point interconnect, a switched interconnect, or other techniques, may be used. In at least one embodiment, graphics processor 1208 is coupled to ring interconnect 1212 via an I / O link 1213.

[0095] In at least one embodiment, I / O link 1213 represents at least one of several versions of I / O interconnects, including an in-chassis I / O interconnect that enables communication between various processor components and an embedded high-performance memory module 1218, such as an eDRAM module. In at least one embodiment, each of the processor cores 1202A-1202N and the graphics processor 1208 uses embedded memory modules 1218 as a shared last-level cache.

[0096] In at least one embodiment, processor cores 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in the instruction set architecture (ISA) sense, where one or more of processor cores 1202A-1202N execute a common instruction set, while one or more other cores of processor cores 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in the microarchitectural sense, where one or more cores having relatively higher power consumption are coupled with one or more power cores having lower power consumption. In at least one embodiment, processor 1200 may be implemented on one or more chips or as an integrated circuit (SoC).

[0097] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, portions of or all of the inference and / or training logic 715 may be integrated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein may utilize one or more of the ALUs included in a graphics processor 1512, one or more graphics cores 1202A-1202N, or other components in Fig. 12. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that described in Fig. 7A or Fig. 7B. In at least one embodiment, weighting parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of a graphics processor 1200 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0098] Such components can be used to determine one or more emotion values ​​from audio data. VIRTUALIZED COMPUTING PLATFORM

[0099] Fig. 13 is an example data flow diagram for a process 1300 for creating and deploying an image processing and inferencing pipeline, according to at least one embodiment. In at least one embodiment, the process 1300 may be provided for use with imaging devices, processing devices, and / or other device types in one or more facilities 1302. The process 1300 may be executed in a training system 1304 and / or in a deployment system 1306. In at least one embodiment, the training system 1304 may be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object recognition algorithms, computer vision algorithms, etc.) for use in the deployment system 1306.In at least one embodiment, deployment system 1306 may be configured to offload processing and computing resources among a distributed computing environment to reduce infrastructure requirements at facility 1302. In at least one embodiment, one or more applications in a pipeline may use or consume services (e.g., inference, visualization, compute, AI, etc.) of deployment system 1306 during application execution.

[0100] In at least one embodiment, some of the applications used in advanced processing and inference pipelines may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning models may be trained at device 1302 using imaging or sequencing data 1308 generated at device 1302 (and stored in one or more picture archiving and communication system (PACS) servers at device 1302), trained using imaging or sequencing data 1308 from one or more other devices, or a combination thereof.In at least one embodiment, the training system 1304 may be used to provide applications, services, and / or other resources for generating working, deployable machine learning models for the deployment system 1306.

[0101] In at least one embodiment, a model registry 1324 may be secured by object storage that may support versioning and object metadata. In at least one embodiment, the object storage may be secured, for example, by application programming interface (API) compatible cloud storage (e.g., the cloud 1426 of Fig. 14) be accessible from within a cloud platform. In at least one embodiment, the machine learning models within the model registry 1324 may be listed, modified, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API may provide access to methods that allow users with appropriate credentials to associate models with applications, so that models may be executed as part of the execution of containerized instantiations of applications.

[0102] In at least one embodiment, the training pipeline 1404 ( Fig. 14) may include a scenario where device 1302 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by one or more imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 1308 is received, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to the imaging data 1308 to be used as ground truth data for a machine learning model. In at least one embodiment, AI-assisted annotation 1310 may include one or more machine learning models (e.g.,convolutional neural networks (CNNs), which can be trained to generate annotations corresponding to certain types of imaging data 1308 (e.g., from certain devices). In at least one embodiment, the AI-assisted annotations 1310 can then be used directly or can be adjusted or fine-tuned using an annotation tool to generate baseline reference data. In at least one embodiment, AI-assisted annotations 1310, labeled clinical data 1312, or a combination thereof can be used as baseline reference data for training a machine learning model. In at least one embodiment, a trained machine learning model can be referred to as an output model 1316 and used by the deployment system 1306 as described herein.

[0103] In at least one embodiment, a training pipeline 1404 ( Fig. 14) may include a scenario where device 1302 requires a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 1306, wherein device 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from model registry 1324. In at least one embodiment, model registry 1324 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 1324 may be trained on imaging data from devices other than device 1302 (e.g.,remotely located facilities). In at least one embodiment, the machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a specific location, the training may take place at that location or at least in a manner that protects the confidentiality of the imaging data or restricts the imaging data from being transferred out of the facility. Once a model is trained or partially trained at a location, in at least one embodiment, a machine learning model may be added to the model registry 1324.In at least one embodiment, a machine learning model may then be retrained or updated at any number of other facilities, and a retrained or updated model may be made available in model registry 1324. In at least one embodiment, a machine learning model may then be selected from model registry 1324 and designated as output model 1316 and used in deployment system 1306 to perform one or more processing tasks for one or more deployment system applications.

[0104] In at least one embodiment, the training pipeline 1404 ( Fig. 14) include a scenario of device 1302 requiring a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 1306, but device 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model registry may not be fine-tuned or optimized for imaging data 1308 generated at device 1302 due to differences in populations, the robustness of training data used to train a machine learning model, the diversity of training data anomalies, and / or other issues with the training data.In at least one embodiment, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to imaging data 1308 to be used as baseline reference data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 may be used as baseline reference data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training 1314. In at least one embodiment, model training 1314, e.g., AI-assisted annotations 1310, labeled clinical data 1312, or a combination thereof, may be used as baseline reference data for retraining or updating a machine learning model.In at least one embodiment, a trained machine learning model may be referred to as output model 1316 and used by deployment system 1306 as described herein.

[0105] In at least one embodiment, deployment system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, deployment system 1306 may include a software "stack," such that software 1318 may be built on top of services 1320 and may use services 1320 to perform some or all of the processing tasks and services 1320, and software 1318 may be built on top of hardware 1322 and may use hardware 1322 to perform processing, storage, and / or other computational tasks of deployment system 1306. In at least one embodiment, software 1318 may include any number of different containers, each container capable of executing an instantiation of an application.In at least one embodiment, each application may perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inferencing, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline may be defined based on selections of various containers desired or required for processing imaging data 1308, in addition to containers that receive and configure imaging data for use by each container and / or for use by device 1302 after processing via a pipeline (e.g., to convert outputs back into a usable data type).In at least one embodiment, a combination of containers within software 1318 (e.g., forming a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and a virtual instrument may leverage services 1320 and hardware 1322 to perform some or all of the processing tasks of applications instantiated in containers.

[0106] In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data may be representative of one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may undergo preprocessing as part of the data processing pipeline to prepare data for processing by one or more applications.In at least one embodiment, post-processing may be performed on an output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, the inference tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models 1316 of the training system 1304.

[0107] In at least one embodiment, the tasks of the data processing pipeline may be encapsulated in container(s), each representing a discrete, fully functioning instantiation of an application and a virtual computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications may be published in a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in the model registry 1324 and associated with one or more applications. In at least one embodiment, images of applications (e.g.,Container images) must be available in a container registry, and once selected by a user from a container registry for use in a pipeline, an image can be used to create a container for an instantiation of an application for use by a user's system.

[0108] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) may develop, publish, and store applications (e.g., as containers) for performing image processing and / or inference on supplied data. In at least one embodiment, the development, publishing, and / or storage may be performed using a software development kit (SDK) associated with a system (e.g., to ensure that a developed application and / or container is compliant or compatible with a system). In at least one embodiment, an application that is developed may be deployed locally (e.g., in a first device to data from a first device) with an SDK that may support at least some of the services 1320, as a system (e.g., as a system 1400 of Fig. 14). In at least one embodiment, because DICOM objects may contain any number from one to hundreds of images or other data types, and due to data variation, a developer may be responsible for managing (e.g., setting constructs for, incorporating preprocessing into an application, etc.), extraction, and preparation of incoming data. In at least one embodiment, once an application has been validated by system 1400 (e.g., for accuracy), it may be available in a container registry for selection and / or implementation by a user to perform one or more processing tasks with respect to data at a user's device (e.g., a second device).

[0109] In at least one embodiment, developers may then deploy applications or containers over a network for access and use by users of a system (e.g., the system 1400 of Fig. 14). In at least one embodiment, completed and validated applications or containers may be stored in a container registry, and associated machine learning models may be stored in the model registry 1324. In at least one embodiment, a requesting entity providing an inference or image processing request may search a container registry and / or the model registry 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements for inclusion in the data processing pipeline, and submit an imaging processing request.In at least one embodiment, a request may include input data (and associated patient data in some examples) necessary to perform a request and / or may include a selection of one or more applications and / or machine learning models to be executed in processing a request. In at least one embodiment, a request may then be passed to one or more components of the deployment system 1306 (e.g., a cloud) to perform processing of the data processing pipelines. In at least one embodiment, processing by the deployment system 1306 may include referencing selected items (e.g., applications, containers, models, etc.) from a container registry and / or the model registry 1324. In at least one embodiment, once the results are generated by a pipeline, the results may be provided to a user for reference (e.g.,for visualization in a visualization application package running on a local on-site workstation or end device).

[0110] In at least one embodiment, to assist in the processing or execution of applications or containers in pipelines, services 1320 may be leveraged. In at least one embodiment, services 1320 may include compute services, artificial intelligence (AI) services, visualization services, and / or other types of services. In at least one embodiment, services 1320 may provide functionality common to one or more applications in software 1318, such that the functionality may be abstracted to a service that may be invoked or leveraged by the applications. In at least one embodiment, the functionality provided by services 1320 may be executed more dynamically and efficiently while also scaling well by allowing applications to process data in parallel (e.g., using a parallel computing platform 1430 ( Fig. 14)). In at least one embodiment, instead of each application sharing a similar functionality offered by service 1320 having to have a respective instance of service 1320, service 1320 may be shared between and among different applications. In at least one embodiment, the services may include an inference server or inference engine that may be used to perform, as non-limiting examples, detection or segmentation tasks. In at least one embodiment, a model training service may be included to train and / or retrain machine learning models. In at least one embodiment, a data augmentation service may be further included that enables the extraction, resizing, scaling, and / or other augmentation of GPU-accelerated data (e.g.,DICOM data, RIS data, CIS data, RESTful data, RPC data, raw data, etc.). In at least one embodiment, a visualization service may be used that can add image rendering effects, such as ray tracing, rasterization, denoising, sharpening, etc., to make two-dimensional (2D) and / or three-dimensional (3D) models more realistic. In at least one embodiment, virtual instrument services may be included that provide beamforming, segmentation, inferencing, imaging, and / or support for other applications within the virtual instrument pipelines.

[0111] In at least one embodiment where a service 1320 comprises an AI service (e.g., an inference service), one or more machine learning models may be executed by invoking an inference service (e.g., an inference server) (e.g., as an API call) to execute machine learning models or their processing as part of application execution. In at least one embodiment where another application comprises one or more machine learning models for segmentation tasks, an application may invoke an inference service to execute machine learning models to perform one or more processing operations associated with segmentation tasks.In at least one embodiment, the software 1318 implementing the advanced processing and inference pipeline including a segmentation application and anomaly detection application may be streamlined because each application may call the same inference service to perform one or more inference tasks.

[0112] In at least one embodiment, hardware 1322 may include GPUs, CPUs, DPUs, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, various types of hardware 1322 may be used to provide efficient, purpose-built support for software 1318 and services 1320 in deployment system 1306. In at least one embodiment, the use of GPU processing may be implemented for local processing (e.g., at device 1302), in an AI / deep learning system, in a cloud system, and / or in other processing components of deployment system 1306 to improve the efficiency, accuracy, and effectiveness of image processing and generation.In at least one embodiment, software 1318 and / or services 1320 may be optimized for GPU processing related to deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least some of the computing environment of deployment system 1306 and / or training system 1304 may be executed in a data center, on one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of NVIDIA's DGX system). In at least one embodiment, hardware 1322 may include any number of GPUs that may be invoked to perform processing of data in parallel, as described herein.In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be executed using one or more AI / supercomputer deep learning and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering system or an orchestration system (e.g., KUBERNETES) across multiple GPUs to allow seamless scaling and load balancing.

[0113] Fig. 14 is a system diagram for an exemplary system 1400 for creating and deploying an imaging deployment pipeline, according to at least one embodiment. In at least one embodiment, the system 1400 may be used to implement the process 1300 of Fig. 13 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, system 1400 may include training system 1304 and deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described herein.

[0114] In at least one embodiment, system 1400 (e.g., training system 1304 and / or deployment system 1306) may be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 may be implemented locally with respect to a healthcare provider organization or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in cloud 1426 may be restricted to authorized users through mandated security measures or protocols. In at least one embodiment, a security protocol may include web tokens, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization.In at least one embodiment, the APIs of virtual instruments (described herein) or other instantiations of system 1400 may be restricted to a set of public IPs that have been verified or authorized for interaction.

[0115] In at least one embodiment, various components of system 1400 may communicate using one of a variety of different network types, including local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of system 1400 (e.g., for transmitting inference requests, for receiving results of inference requests, etc.) may be communicated via one or more data buses, wireless data protocols (WiFi), wired data protocols (e.g., Ethernet), etc.

[0116] In at least one embodiment, the training system 1304 may execute training pipelines 1404 similar to those described herein with reference to Fig. 13. In at least one embodiment, where one or more machine learning models are to be used in the deployment pipelines 1410 by the deployment system 1306, training pipelines 1404 may be used to train or retrain one or more (e.g., pre-trained) models, and / or to implement one or more pre-trained models 1406 (e.g., without requiring retraining or updating). In at least one embodiment, one or more output models 1316 may be generated as a result of the training pipelines 1404. In at least one embodiment, the training pipelines 1404 may include any number of processing steps, such as, but not limited to, transforming or adapting imaging data (or other input data).In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the deployment system 1306. In at least one embodiment, the training pipeline 1404 may be similar to one described with reference to FIG. Fig. 13 is to be used for a first machine learning model, the training pipeline 1404, which is similar to one with reference to Fig. 13, can be used for a second machine learning model, and the training pipeline 1404, which is similar to one with respect to Fig. 13 may be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 may be used depending on what is required for each particular machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment, such that the machine learning models do not receive any processing by the training system 1304 and may be implemented by the deployment system 1306.

[0117] In at least one embodiment, one or more output models 1316 and / or pre-trained models 1406 may include any type of machine learning model depending on the implementation or embodiment. In at least one embodiment, and without limitation, the machine learning models used by system 1400 may include one or more machine learning models that include linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forest,

[0118] Use dimensionality reduction algorithms, gradient boosting algorithms, one or more neural networks (e.g., autoencoders, convolutional algorithms, recurrent algorithms, perceptrons, Long / Short Term Memory (LSTM), Hopfield, Boltzmann, Deep Belief, Deconvolutional, Generative Adversarial and / or Liquid State Machine, etc.) and / or other types of machine learning models.

[0119] In at least one embodiment, training pipelines 1404 may include AI-assisted annotation, as described here at least with respect to Fig. 15B. In at least one embodiment, labeled data 1312 (e.g., conventional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a CAD (computer-aided design) program, a labeling program, another type of program suitable for generating ground truth annotations or labels, and / or, in some examples, hand-drawn. In at least one embodiment, the ground truth data may be synthetically generated (e.g., from computer models or renderings), physically generated (e.g., designed and generated from real data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g.,a labeling or annotation expert determines the location of the markers) and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data type used by machine learning models), there may be corresponding ground truths generated by the training system 1304. In at least one embodiment, the AI-assisted annotation may be performed as part of the deployment pipelines 1410, either in addition to or instead of the AI-assisted annotation included in the training pipelines 1410. In at least one embodiment, the system 1400 may include a multi-layer platform including an application layer (e.g., software 1318) of diagnostic applications (or other application types) that can perform one or more medical imaging and diagnostic functions. In at least one embodiment, the system 1400 may be communicatively (e.g.,via encrypted connections) to PACS server networks of one or more facilities. In at least one embodiment, system 1400 may be configured to access and reference data from PACS servers to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0120] In at least one embodiment, a software layer may be implemented as a secure, encrypted, and / or authenticated API through which applications or containers may be invoked from one or more external environments (e.g., device 1302). In at least one embodiment, applications may then invoke or execute one or more services 1320 for performing computational, AI, or visualization tasks associated with the respective applications, and the software 1318 and / or services 1320 may leverage the hardware 1322 to perform processing tasks in an effective and efficient manner.

[0121] In at least one embodiment, deployment system 1306 may execute deployment pipelines 1410. In at least one embodiment, deployment pipelines 1410 may include any number of applications that may be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other types of data) generated by imaging devices, sequencing devices, genomics devices, etc., including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline 1410 for an individual device may be referred to as a virtual device for a device (e.g., a virtual ultrasound device, a virtual CT scan device, a virtual sequencing device, etc.). In at least one embodiment, there may be more than one deployment pipeline 1410 for a single device, depending on what information is desired from the data generated by a device.At least in one embodiment where detecting abnormalities from an MRI machine is desired, there may be a first deployment pipeline 1410, and where image enhancement from the output of an MRI machine is desired, there may be a second deployment pipeline 1410.

[0122] In at least one embodiment, an application for generating images may include a processing task that includes the use of a machine learning model. In at least one embodiment, a user may wish to use their own machine learning model or select a machine learning model from the model registry 1224. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model to include in an application for performing a processing task. In at least one embodiment, applications may be selected and customized, and by defining constructs of applications, the deployment and implementation of applications for a particular user is presented as a more seamless user experience.In at least one embodiment, deployment pipelines 1410 may be even more user-friendly, provide easier integration, and deliver more accurate, efficient, and timely results by leveraging other features of system 1400, such as services 1320 and hardware 1322.

[0123] In at least one embodiment, deployment system 1306 may include a user interface 1414 (UI) (e.g., a graphical user interface, a web interface, etc.) that may be used to select applications for inclusion in deployment pipeline(s) 1410, arrange, modify, or change applications or parameters or constructs thereof, use and interact with deployment pipeline(s) 1410 during setup and / or deployment, and / or otherwise interact with deployment system 1306. In at least one embodiment, although not illustrated with respect to deployment system 1304, user interface 1414 (or another user interface) may be used to select models for use in deployment system 1306, select models for retraining in deployment system 1304, and / or otherwise interact with deployment system 1304.

[0124] In at least one embodiment, the pipeline manager 1412 may be used in addition to an application orchestration system 1428 to manage the interaction between applications or containers of the deployment pipeline(s) 1410 and services 1320 and / or the hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate application-to-application, application-to-service 1320, and / or application or service-to-hardware 1322 interactions. In at least one embodiment, the pipeline manager 1412, although shown as being included in the software 1318, may be included in the services 1320 (e.g., as shown in Fig. 11). In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline(s) 1410 (e.g., a reconstruction application, a segmentation application, etc.) with individual containers, each application may be executed in a self-contained environment (e.g., at the kernel level) to increase speed and efficiency.

[0125] In at least one embodiment, each application and / or container (or an image thereof) may be developed, modified, and deployed individually (e.g., a first user or developer may develop a first application, and a second user or developer may develop, modify, and deploy a second application separately from a first user or developer), thereby allowing focus and attention to a task of a single application and / or one or more containers without being hindered by tasks of one or more other applications or containers. In at least one embodiment, communication and cooperation between different containers or applications may be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on application or container constructs), the application orchestration system 1428 and / or the pipeline manager 1412 may facilitate communication among and between, and resource sharing among, each of the applications or containers. In at least one embodiment, since one or more of the applications or one or more of the containers in the deployment pipeline(s) 1410 may share the same services and resources, the application orchestration system 1428 may orchestrate, load balance, and determine sharing between and among different applications or containers.In at least one embodiment, a scheduler may be used to track resource requests from applications or containers, the current or planned usage of those resources, and resource availability. Thus, in at least one embodiment, a scheduler may allocate resources to different applications and distribute resources between and among applications in light of system requirements and availability. In some examples, a scheduler (and / or another component of the application orchestration system 1428) may determine the availability and distribution of resources based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), urgency of the need for data output (e.g., to determine whether to perform real-time or deferred processing), etc.

[0126] In at least one embodiment, services 1320 leveraged and shared by applications or containers in the deployment system 1306 may include compute services 1416, AI services 1418, visualization services 1420, and / or other types of services. In at least one embodiment, applications may invoke (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, compute services 1416 may be leveraged by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1416 may be leveraged to enable parallel processing (e.g.,using a parallel computing platform 1430) for processing data by one or more of the applications and / or one or more tasks of a single application substantially concurrently. In at least one embodiment, the parallel computing platform 1430 (e.g., NVIDIA's CUDA) may allow universal computing on GPUs (e.g., GPUs 1422). In at least one embodiment, a software layer of a parallel computing platform 1430 may provide access to virtual instruction sets and parallel compute elements of GPUs to execute compute cores. In at least one embodiment, the parallel computing platform 1430 may include memory, and in some embodiments, memory may be shared between and among multiple containers and / or between and among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared memory segment of a parallel computing platform 1430 (e.g., where multiple different stages of an application or multiple applications process the same information). In at least one embodiment, instead of making a copy of the data and moving data to different locations in memory (e.g., a read / write operation), data at the same memory location may be used for any number of processing tasks (e.g., concurrently, at different times, etc.).In at least one embodiment, since data is used to generate new data as a result of processing, this information about a new data location can be stored and shared between different applications. In at least one embodiment, the location of the data and a location of updated or modified data can be part of a definition of how a payload is understood within containers.

[0127] In at least one embodiment, AI services 1418 may be leveraged to perform inference services for executing one or more machine learning models associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI services 1418 may leverage an AI system 1424 to execute one or more machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature recognition, classification, and / or other inference tasks. In at least one embodiment, applications of deployment pipeline(s) 1410 may use one or more of the output models 1316 from the training system 1304 and / or other application models to perform inference on imaging data.In at least one embodiment, two or more examples of inferencing using the application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high priority / low latency path that can achieve higher service level agreements, such as to perform inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard priority path that can be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1428 may allocate resources (e.g.,Services 1320 and / or hardware 1322) based on priority paths for different inference tasks of AI services 1418.

[0128] In at least one embodiment, shared storage for AI services 1418 may be deployed in system 1400. In at least one embodiment, the shared storage may operate as a cache (or other type of storage device) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a request may be received by a set of API instances of deployment system 1306, and one or more instances may be selected (e.g., for best fit, load balancing, etc.) to process a request.In at least one embodiment, to process a request, a request may be entered into a database, a machine learning model may be located by the model registry 1324 if it is not already in a cache, a validation step may ensure that a corresponding machine learning model is loaded into a cache (e.g., shared storage), and / or a copy of a model may be stored in a cache. In at least one embodiment, a scheduler (e.g., of the pipeline manager 1412) may be used to launch an application referenced by a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server has not already been started to execute a model, an inference server may be started.Any number of inference servers can be started per model. In at least one embodiment, in a pull model where inference servers are clustered, models can be cached whenever load balancing is advantageous. In at least one embodiment, inference servers can be statically loaded into corresponding, distributed servers.

[0129] In at least one embodiment, inference may be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server may be associated with a model (and optionally multiple versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance may be loaded. In at least one embodiment, a model may be passed to an inference server upon starting an inference server, so that the same container may be used to serve different models, as long as the inference server runs as a different instance.

[0130] In at least one embodiment, during application execution, an inference request for a given application may be received, and a container (e.g., hosting an instance of an inference server) may be loaded (if not already loaded), and a startup procedure may be invoked. In at least one embodiment, preprocessing logic in a container may load incoming data, decode it, and / or perform any additional preprocessing on the data (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, a container may perform inference on data as needed. In at least one embodiment, this may involve a single inference call on one image (e.g., a hand x-ray) or may require inference on hundreds of images (e.g., a chest CT).In at least one embodiment, an application may summarize the results before completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating a visualization, or generating text to summarize findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have a real-time priority (turnaround time (TAT) < 1 min), while others may have a lower priority (e.g., turnaround time < 10 min). In at least one embodiment, model execution times may be measured by the requesting institution or entity and may include time to traverse the partner network as well as execution on an inference service.

[0131] In at least one embodiment, the transfer of requests between services 1320 and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provided via a queue. In at least one embodiment, a request is placed in a queue via an API for a custom application / tenant ID combination, and an SDK fetches a request from a queue and passes the request to an application. In at least one embodiment, a queue name may be provided in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication through a queue may be useful because it allows each instance of an application to begin work as it becomes available.The results may be propagated back through a queue to ensure that no data is lost. In at least one embodiment, queues may also provide the ability to segment work, as highest-priority work may go into a queue to which most instances of an application are connected, while lowest-priority work may go into a queue to which a single instance is connected and processes the tasks in a received order. In at least one embodiment, an application may run on a GPU-accelerated instance spawned in the cloud 1426, and an inference service may perform inferencing on a GPU.

[0132] In at least one embodiment, visualization services 1420 may be leveraged to generate visualizations for visualizing outputs from applications and / or from one or more deployment pipelines 1410. In at least one embodiment, GPUs 1422 of visualization services 1420 may be leveraged to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, may be implemented by the visualization services 1420 to generate higher quality visualizations. In at least one embodiment, visualizations may include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstructions, 2D tomography slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments may be used to create a virtual interactive display or environment (e.g.,a virtual environment) for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, the visualization services 1420 may include an internal visualizer, kinematics, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0133] In at least one embodiment, hardware 1322 may include GPUs 1422, an AI system 1424, a cloud 1426, and / or any other hardware used to execute a training system 1304 and / or a deployment system 1306. In at least one embodiment, GPUs 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that may be used to perform processing tasks of compute services 1416, AI services 1418, visualization services 1420, other services, and / or any of the features or functionality of software 1318. For example, with respect to AI services 1418, the GPUs 1422 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on outputs of machine learning models, and / or inferencing (e.g.,to execute machine learning models). In at least one embodiment, cloud 1426, AI system 1424, and / or other components of system 1400 may use GPUs 1422. In at least one embodiment, cloud 1426 may comprise a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 may use GPUs, and cloud 1426—or at least a portion tasked with deep learning or inferencing—may be executed using one or more AI systems 1424. Therefore, although hardware 1322 is illustrated as discrete components, this is not intended to be limiting, and any components of hardware 1322 may be combined or leveraged with any other components of hardware 1322.

[0134] In at least one embodiment, the AI ​​system 1424 may include a purpose-built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that may be executed using one or more GPUs 1422 in addition to the CPUs, RAM, storage, and / or other components, features, or functionality. In at least one embodiment, one or more AI systems 1424 may be implemented in a cloud 1426 (e.g., in a data center) for performing one or all of the AI-based processing tasks of the system 1400.

[0135] In at least one embodiment, cloud 1426 may include a GPU-accelerated infrastructure (e.g., NGC from NVIDIA) that may provide a GPU-optimized platform for running processing tasks of system 1400. In at least one embodiment, cloud 1426 may include AI system(s) 1424 for performing one or more of the AI-based tasks of system 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1426 may be integrated with application orchestration system 1428 and leverage multiple GPUs to allow seamless scaling and load balancing between and among applications and services 1320. In at least one embodiment, cloud 1426 may be tasked with running at least some of the services 1320 of system 1400, including compute services 1416, AI services 1418, and / or visualization services 1420, as described herein.In at least one embodiment, the cloud 1426 may perform small and large batch inference (e.g., running NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run the application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality kinematics), and / or may provide other functionality for the system 1400.

[0136] Fig. 15A illustrates a data flow diagram for a process 1500 to train, retrain, or update a machine learning model, according to at least one embodiment. In at least one embodiment, the process 1500 may be performed using, as a non-limiting example, the system 1400 of Fig. 14. In at least one embodiment, process 1500 may leverage services 1320 and / or hardware 1322 of system 1400, as described herein. In at least one embodiment, refined models 1512 generated by process 1500 may be executed by a deployment system 1306 for one or more containerized applications in deployment pipelines 1410.

[0137] In at least one embodiment, model training 1314 may include retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as a customer record 1506 and / or new baseline reference data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, an output or loss layer(s) of the initial model 1504 may be reset or deleted and / or replaced with an updated or new output or loss layer(s). In at least one embodiment, the initial model 1504 may have previously fine-tuned parameters (e.g.,weights and / or biases) left over from the previous training, so that training or retraining 1314 does not take as long or require as much processing as training a model from scratch. In at least one embodiment, during model training 1314, by resetting or replacing the output or loss layer(s) of the initial model 1504, the parameters for a new dataset may be adjusted based on loss calculations that match the accuracy of the output or loss layer(s) in generating predictions on a new customer dataset 1506 (e.g., image data 1308 from . Fig. 13) are updated and re-aligned.

[0138] In at least one embodiment, pre-trained models 1406 may be stored in a data store or registry (e.g., the model registry 1324 of Fig. 13). In at least one embodiment, the pre-trained models 1406 may have been trained, at least in part, in one or more facilities other than a facility executing the process 1500. In at least one embodiment, to protect the privacy and rights of patients, subjects, or customers of different facilities, the pre-trained models 1406 may have been trained on-site using customer or patient data generated on-site. In at least one embodiment, the pre-trained models 1406 may be trained using the cloud 1426 and / or other hardware 1322, but no confidential, privacy-protected patient data may be transmitted to, used by, or accessible by any components of the cloud 1426 (or other external hardware).In at least one embodiment where a pre-trained model 1406 is trained to use patient data from more than one facility, the pre-trained model 1406 may have been individually trained for each facility before being trained with patient or customer data from another facility. In at least one embodiment, such as when customer or patient data has been exempted from privacy concerns (e.g., through a waiver, for experimental use, etc.) or when customer or patient data is included in a public dataset, customer or patient data from any number of facilities may be used to train the pre-trained model 1406 on and / or off-premises, such as in a data center or other cloud computing infrastructure.

[0139] In at least one embodiment, when selecting applications for use in the deployment pipelines 1410, a user may also select machine learning models to be used for specific applications. In at least one embodiment, a user may not have a model to use, so a user may select a pre-trained model 1406 for use with an application. In at least one embodiment, the pre-trained model 1406 may not be optimized to produce accurate results on the customer data set 1506 of a user's facility (e.g., based on patient diversity, demographic data, types of medical imaging devices used, etc.).In at least one embodiment, prior to deploying a pre-trained model 1406 into a deployment pipeline 1410 for use with an application(s), the pre-trained model 1406 may be updated, retrained, and / or fine-tuned for use in a particular device.

[0140] In at least one embodiment, a user may select the pre-trained model 1406 to be updated, retrained, and / or fine-tuned, and this pre-trained model 1406 may be referred to as the initial model 1504 for the training system 1304 within the process 1500. In at least one embodiment, a customer data set 1506 (e.g., imaging data, genomics data, sequencing data, or other types of data generated by devices in a facility) may be used to perform model training 1314 (which may include, without limitation, transfer learning) on ​​the initial model 1504 to generate a refined model 1512. In at least one embodiment, baseline reference data corresponding to the customer data set 1506 may be generated by the training system 1304.In at least one embodiment, basic reference data may be generated at least in part by clinicians, scientists, physicians, general practitioners in an institution (e.g., as labeled clinical data 1312 of . Fig. 13) are generated.

[0141] In at least one embodiment, AI-assisted annotation 1310 may be used to generate baseline reference data, in some examples. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) may leverage machine learning models (e.g., neural networks) to generate suggested or predicted baseline reference data for a customer record. In at least one embodiment, user 1510 may use annotation tools in a user interface (a graphical user interface (GUI)) on computing device 1508.

[0142] In at least one embodiment, user 1510 may interact with a GUI via computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, a polygon editing feature may be used to move the vertices of a polygon to multiple precise or fine-tuned locations.

[0143] In at least one embodiment, once the customer data set 1506 has associated baseline reference data, the baseline reference data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training 1314 to generate the refined model 1512. In at least one embodiment, the customer data set 1506 may be applied to the initial model 1504 any number of times, and baseline reference data may be used to update the parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, once the refined model 1512 has been generated, the refined model 1512 may be deployed within one or more deployment pipelines 1410 in a device for performing one or more processing tasks with respect to medical imaging data.

[0144] In at least one embodiment, the refined model 1512 may be uploaded to pre-trained models 1406 in a model registry 1324 for selection by another entity. In at least one embodiment, this process may be completed with any number of entities, allowing the refined model 1512 to be further refined on new datasets as often as desired to produce a more universal model.

[0145] Fig. 15B is an example illustration of a client-server architecture 1532 for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment. In at least one embodiment, AI-assisted annotation tools 1536 may be instantiated based on a client-server architecture 1532. In at least one embodiment, annotation tools 1536 may, for example, assist radiologists in imaging applications in identifying organs and abnormalities. In at least one embodiment, the imaging applications may include software tools that assist the user 1510, as a non-limiting example, in identifying some extreme points on a particular organ of interest in raw images 1534 (e.g., in a 3D MRI or CT scan) and automatically receive annotated results for all 2D slices of a particular organ.In at least one embodiment, the results may be stored in a data store as training data 1538 and used (by way of example and without limitation) as baseline reference data for training. In at least one embodiment, when computing device 1508 sends extreme points for AI-assisted annotation 1310, a deep learning model, for example, may receive this data as input and return inference results of a segmented organ or anomaly. In at least one embodiment, pre-instantiated annotation tools, such as AI-assisted annotation tool 1536B in FIG. Fig.15B, may be enhanced by making API calls (e.g., an API call 1544) to a server, such as an annotation assistance server 1540, which may include a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, an annotation model registry may store pre-trained models 1542 (e.g., machine learning models such as deep learning models) that are pre-trained to perform AI-assisted annotation on a particular organ or anomaly. These models may be further updated using training pipelines 1404. In at least one embodiment, pre-installed annotation tools may be enhanced over time as new labeled clinical data 1312 is added.

[0146] Such components can be used to determine one or more emotion values ​​from audio data.

[0147] Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof are shown in the drawings and have been described in detail above. It should be understood, however, that there is no intention to limit the disclosure to any specific form or forms disclosed; on the contrary, the invention is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure as defined by the appended claims.

[0148] The use of the terms "a" and "an" and "the" or similar references in the context of describing the disclosed embodiments (particularly in the context of the following claims) is to be interpreted to cover both singular and plural, unless otherwise stated herein or clearly contradicted by the context. The terms "comprising," "having," and "include" are to be interpreted as open-ended terms (i.e., "including but not limited to") unless otherwise stated. The term "connected," when unmodified and referring to physical connections, is to be interpreted as partially or wholly contained in, attached to, or coupled to one another, even if something intervening.Citation of ranges of values ​​herein is intended merely as a shorthand method of individually referring to each value falling within that range, unless otherwise noted herein, and each value is incorporated into the specification as if individually cited herein. Use of the term "set" (e.g., "a set of elements") or "subset" is, unless otherwise stated or contradicted by the context, to mean a non-empty collection having one or more elements. Furthermore, unless otherwise stated or contradicted by the context, the term "subset" of a corresponding set does not necessarily denote a proper subset of a corresponding set; rather, the subset and the corresponding set may be the same.

[0149] Conjunctive language, such as phrases in the form of "at least one of A, B, and C" or "at least one of A, B, and C," on the other hand, unless explicitly stated otherwise or otherwise clearly contradicted by the context, is to be understood with the context as being generally used to present that an element, term, etc., is either A or B or C, or any non-empty subset of the set of A, B, and C. For example, in an illustrated example of a set with three elements, the conjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" involve reference to each of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such conjunctive language is not generally intended to imply that certain embodiments require that at least one of A, at least one of B, and at least one of C be present.Additionally, unless otherwise stated or contradicted by context, the term "plural" indicates a plural state (e.g., "a plurality of elements" indicates multiple elements). A plurality is at least two, but can be more if so indicated either explicitly or by context. Furthermore, unless otherwise stated or clear by context, the phrase "based on" means "based at least in part on" and not "based solely on."

[0150] Operations of processes described herein may be performed in any suitable order, unless otherwise stated herein or clearly contradicted by context. In at least one embodiment, a process such as those processes described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors using hardware, or combinations thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors.In at least one embodiment, the computer-readable medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., a propagating transient electrical or electromagnetic transmission), but does not include transient data storage circuits (e.g., buffers, caches, and queues) within transceivers of transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having stored thereon executable instructions (or other memory to store executable instructions) that, when executed by one or more processors of a computer system (i.e., as a result of being executed), cause the computer system to perform operations described herein.In one embodiment, a set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media of the plurality of non-transitory computer-readable storage media lacks any code at all, while the plurality of non-transitory computer-readable storage media collectively store all code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors.

[0151] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform the processes described herein, and such computer systems are configured with applicable hardware and / or software that enable the operations to be performed. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computer system that includes multiple devices that operate differently such that the distributed computer system performs the operations described herein, and such that no single device performs all of the operations.

[0152] The use of any and all examples or exemplary language provided herein (e.g., "such as") is intended merely to further illustrate embodiments of the disclosure and does not impose any limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed to identify any unclaimed element as essential to practicing the disclosure.

[0153] All references, including publications, patent applications, and patents cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in its entirety herein.

[0154] Throughout the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended to be synonymous with each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are directly or indirectly in physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, yet still cooperate or interact with each other.

[0155] Unless specifically stated otherwise, it will be appreciated that, throughout the specification, terms such as "process," "compute," "calculate," "determine," or the like refer to the action and / or processes of a computer or computer system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities within the computer system's registers and / or memory into other data also represented as physical quantities within the memory, registers, or other such information storage, transmission, or display devices.

[0156] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and transforms that electronic data into other electronic data that can be stored in registers and / or memory. As used herein, "software processes" may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Also, any process may refer to multiple processes for dispatching instructions in sequence or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein in that the system may embody one or more methods, and the methods may be considered a system.

[0157] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data to a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving the data as a parameter of a function call or a call to an application programming interface.In at least one embodiment, processes of obtaining, acquiring, receiving, or inputting analog or digital data may be achieved by transmitting the data over a serial or parallel interface. In at least one embodiment, processes of obtaining, acquiring, receiving, or inputting analog or digital data may be achieved by transmitting the data from the providing entity to the acquiring entity over a computer network. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data may be achieved by transmitting the data as an input or output parameter of a function call, a parameter of a programming interface, or an interprocess communication mechanism.

[0158] Although descriptions herein set forth exemplary embodiments of described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of the disclosure. Furthermore, although specific distributions of responsibilities are defined above for purposes of illustration, various functions and responsibilities could be distributed and allocated in different ways depending on the circumstances.

[0159] Furthermore, although the subject matter has been described in language referring to structural features and / or methodological acts, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, certain features and acts are disclosed as exemplary forms of implementing the claims.

Claims

[1] Computer-implemented method comprising: Calculating, using a transformer-based neural network and based at least in part on audio data representing speech, one or more values ​​indicative of one or more emotions; Determining, based at least in part on the one or more values, that at least one emotion corresponds to the one or more emotions of the language; and Performing one or more operations based at least in part on the at least one emotion. [2] The computer-implemented method of claim 1, wherein the one or more values ​​comprise a respective one or more probability values ​​for each emotion of the one or more emotions, and the one or more values ​​are normalized and summed into an absolute value. [3] The computer-implemented method of claim 1, wherein the one or more emotions comprise at least one of anger, disgust, fear, joy, sadness, or a neutral emotion. [4] A computer-implemented method according to claim 1, further comprising: Providing an interface to receive user input corresponding to one or more settings of the one or more values. [5] A computer-implemented method according to claim 4, further comprising: Receiving, via the interface, one or more emotion strength values ​​for use in weighting the at least one emotion with respect to at least one other emotion of the one or more emotions determined to correspond to the speech. [6] A computer-implemented method according to claim 4, further comprising: Receiving, through the interface, one or more previous values ​​corresponding to the one or more emotions; and Combining the one or more previous values ​​with the one or more values ​​to produce one or more combined values, wherein determining that the at least one emotion corresponds to the one or more emotions of the language is based at least in part on the one or more combined values. [7] A computer-implemented method according to claim 6, further comprising: Receiving one or more previous emotion strength values ​​corresponding to the one or more emotions, wherein the one or more previous emotion strength values ​​indicate one or more weights to be used in combining the one or more previous values ​​with the one or more values. [8] A computer-implemented method according to claim 1, further comprising: Determining one or more probability values ​​for the one or more emotions for each keyframe of a set of keyframes in the audio data, wherein the one or more probability values ​​are determined using a sliding window of audio data for a given audio segment. [9] A computer-implemented method according to claim 8, further comprising: Smoothing the one or more values ​​corresponding to the one or more emotions over multiple iterations. [10] The computer-implemented method of claim 1, wherein the audio data is represented using an audio file format. [11] Processor comprising: one or more processing units to: providing audio data in an audio file format as input to a Transformer neural network; using the Transformer neural network and based at least in part on the audio data, calculate one or more values ​​indicative of one or more emotions corresponding to the audio data; and perform one or more operations based at least in part on a determination that at least one emotion of the one or more emotions corresponds to the audio data. [12] The processor of claim 11, wherein the one or more emotions comprise a predetermined set of emotions, the predetermined set of emotions comprising at least one of anger, disgust, fear, joy, sadness, and a neutral emotion. [13] The processor of claim 11, wherein the one or more processing units further comprise: weight the one or more values ​​based at least in part on one or more emotion intensity values ​​corresponding to the respective emotions of the one or more emotions, wherein the one or more operations are performed based at least in part on the one or more weighted values. [14] The processor of claim 11, wherein the one or more processing units further: receive one or more previous values ​​corresponding to the one or more emotions; and combine the one or more previous values ​​with the one or more values ​​to produce one or more combined values, wherein the determination that the at least one emotion of the one or more emotions corresponds to the audio data is based at least in part on the one or more combined values. [15] The processor of claim 11, wherein the audio file format comprises at least one of an uncompressed audio file format, a lossless compression audio file format, or a lossy compression audio file format. [16] System comprising: one or more processing units to: using a transformer neural network and based at least in part on audio data representing speech, to calculate one or more first values ​​indicating a probability that one or more emotions correspond to the speech; using a neural network and based at least in part on the one or more first values ​​and the audio data, calculate one or more second values ​​indicating one or more positions of one or more feature points corresponding to a virtual object; and render the virtual object based at least in part on the one or more second values. [17] The system of claim 16, wherein the audio data corresponds to an audio file format. [18] The system of claim 16, wherein the audio data is processed into an audio file format using the transformer neural network and the audio data is processed into an image file format using the neural network. [19] The system of claim 16, wherein the one or more feature points correspond to one or more facial features or one or more body features of the virtual object. [20] System according to claim 16, wherein the system comprises at least one of the following: a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system in which one or more virtual machines (VMs) are integrated; a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

Images