Expression of emotions in speech for conversational AI systems and applications

Abstract

In various examples, expression of emotions in speech for conversational AI systems and applications is described. Systems and methods are disclosed that use a machine learning model (e.g., one or more large language models (LLMs)) to determine an emotional state associated with a speech output by a character and one or more values of one or more variables associated with the emotional state and / or the speech. For example, the variables may include the intensity of the emotional state and / or the pitch, speed, volume, accent, and / or the like of the speech. In some examples, the machine learning model may also determine one or more values of the emotional state and / or one or more variables using various types of inputs other than text of speech, such as user data and / or role data. The character may then output the speech in a manner that expresses the emotional state.

Description

BACKGROUND OF THE INVENTION

[0001] Many applications, such as game applications, interactive applications, communication applications, multimedia applications, video conferencing applications, in-vehicle infotainment applications, etc., use an animated character or avatar that interacts with the user of the application / machine / device and / or with other animated characters (e.g., non-player characters (NPCs)) in the application. To provide a more realistic experience to the user, the system may attempt to animate the character by expressing emotions during the interaction with the user. For example, when determining the speech to be output to the user by the animated character, the system may also determine the emotional state associated with the animated character, such as based on an analysis of the text of the speech. The emotional state may then be used such that the animated character outputs the speech in a manner that expresses the emotional state. For example, the voice of the animated character used to output the speech may reflect the emotional state of the animated character.

[0002] However, by using only the text of the speech to determine the emotional state, the system may incorrectly determine the emotional state based on the context of the interaction. For example, people can express the same text, such as "Have a nice day", with different emotional states, such as happy or sad. Thus, by only associating the text with an emotional state and then having the animated character use that emotional state when outputting the speech corresponding to the text, the animated character may use an inappropriate or inaccurate emotional state to express their speech, which may result in an undesirable user experience. Additionally, by using only a set emotional state, such as happy or sad, for the animated character, the system may not be able to have the animated character express a wide range of emotional states through speech. For example, people may express the same emotional state differently at different times, such as whether a person is somewhat happy or very happy. When expressing the same emotional state differently, the speech of the user can also change, such as the characteristics of the user's speech (e.g., pitch, speech rate, etc.). SUMMARY OF THE INVENTION

[0003] Embodiments of the present disclosure relate to expressing emotions in speech for conversational AI systems and applications. Systems and methods are disclosed for using one or more machine learning models to determine an emotional state associated with speech being output by a character and one or more values of one or more variables associated with the emotional state and / or the speech. For example, the one or more variables may include the intensity and / or pitch of the emotional state, speech rate, volume, tone, accent, and / or other attributes of the speech. In some examples, in addition to the text of the speech, the one or more machine learning models may also use various types of input to determine the emotional state and / or one or more values of the one or more variables, such as user data representing information associated with the user and / or character data representing information associated with the character. The systems and methods may then cause the character to output the speech in a manner that expresses the emotional state based at least on the one or more values.

[0004] Contrary to conventional systems, in embodiments, the present systems and methods are capable of using additional input in cooperation with the text of the speech to determine the emotional state associated with the speech. As described in more detail herein, by using the additional input, the current systems can better determine the actual emotional state of the speech - for example, because the same text can be associated with different emotional states based on other circumstances associated with the speech. Additionally, contrary to conventional systems, in some embodiments, the current systems are capable of determining additional values of variables associated with the emotional state and / or the speech. As described in more detail herein, by determining the additional values associated with the variables, the current systems are able to animate the character to better express the emotional state in the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present systems and methods for expressing emotions in speech for conversational AI systems and applications are described in detail below with reference to the drawings, in which:

[0006] Figure 1A A first example data flow diagram of a first process for using one or more machine learning models to generate speech expressing an emotional state in accordance with some embodiments of the present disclosure is shown;

[0007] Figure 1B A second example data flow diagram of a second process for using one or more machine learning models to generate speech expressing an emotional state in accordance with some embodiments of the present disclosure is shown;

[0008] Figure 2 An example of generating text associated with a user input in accordance with some embodiments of the present disclosure is shown;

[0009] Figure 3Shows an example of determining emotion information associated with speech output by a role according to some embodiments of the present disclosure;

[0010] Figure 4 Shows an example of generating speech expressing emotion according to some embodiments of the present disclosure;

[0011] Figure 5 Shows an example of determining text and emotion information associated with speech output by a role according to some embodiments of the present disclosure;

[0012] Figure 6 Shows a data flow diagram of a process for training one or more models to generate emotion information associated with speech according to some embodiments of the present disclosure;

[0013] Figure 7 Shows a flowchart of a method for enabling a role to communicate using speech expressing emotion according to some embodiments of the present disclosure;

[0014] Figure 8 Shows a flowchart of a method for generating audio data representing speech expressing emotion according to some embodiments of the present disclosure;

[0015] Figure 9 Is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0016] Figure 10 Is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Description

[0017] Systems and methods related to expressing emotion in speech for conversational AI systems and applications are disclosed. For example, a system can receive input data associated with at least one of a user or a character being animated. For a user, for example, the input data can include text data representing user input (or converted from audio), audio data representing speech from the user (e.g., in the form of a spectrogram), image data representing an image depicting the user, profile data representing information about the user, and / or any other type of data. As described herein, text can represent one or more letters, words, symbols, numbers, characters, punctuation marks, tokens, and / or the like. Additionally, for a character, the input data can represent characteristics associated with the character (e.g., occupation, relationship, personality traits, etc.), past communications, the current environment (e.g., current interactions with other characters, current location, current goals, etc.), and / or any other information associated with the character. While these examples describe the input data as being associated with a user and / or a character, in other examples, the input data can include any other type of input data (e.g., a prompt, which will be described in more detail herein).

[0018] The system can then use one or more machine learning models associated with generating text (referred to as "first machine learning models" in some examples) to process the input data. For example, one or more first machine learning models can be trained to process the input data and, at least based on that processing, generate text associated with the speech that the character will output. For a first example, if the input data represents input text and / or user speech associated with a query, the text generated by the first machine learning model can be associated with a response to the query. For a second example, if the input data represents character information, such as the current environment associated with the character (e.g., who the character is interacting with), the text generated by one or more first machine learning models can be associated with the current environment.

[0019] The system can also use one or more machine learning models (referred to as "second machine learning models" in some examples) associated with determining emotion information (such as an emotional state) to process the input data and / or text data representing text. As described herein, an emotional state can include, but is not limited to, anger, calm, disgust, fear, happiness, helpfulness, humor, sadness, and / or any other emotional state. In some examples, one or more second machine learning models can be the same as one or more first machine learning models. For example, the system can use a machine learning model trained to both determine text associated with speech and determine emotion information associated with the speech to process the input data. In some examples, the second machine learning model can be different from the first machine learning model. For example, the system can apply the input data and the text data generated using the first machine learning model to the second machine learning model.

[0020] As described herein, the second machine learning model can be trained to determine an emotional state associated with a character and one or more values (e.g., additional emotion information) of one or more variables associated with the emotional state and / or speech. For a first example, if one or more variables are associated with the intensity of an emotional state, the value can indicate very low, low, medium, high, very high, and / or any other intensity level. For a second example, one or more variables associated with speech can include, but are not limited to, volume, pitch, resonance, clarity, speech rate, accent, and / or any other characteristic or attribute associated with speech. Thus, the value of volume can indicate silent, ultra-low, low, medium, high, ultra-high, and / or the like. Additionally, the values associated with pitch and / or speech rate can indicate ultra-low, low, medium, high, ultra-high, and / or the like.

[0021] The system can then apply the text data representing text and / or data representing emotion information (e.g., an emotional state and / or one or more values of one or more variables) (referred to as "emotion data" in some examples) to one or more machine learning models (referred to as "third machine learning models" in some examples) associated with generating speech. For example, one or more third machine learning models can include a text-to-speech model that is trained to generate audio data representing speech. As described herein, the third machine learning model can also be trained to generate audio data such that the speech expresses an emotional state. For a first example, the speech represented by the audio data can be expressed at least based on the intensity of the emotional state. For a second example, the speech represented by the audio data can be associated with one or more values of one or more characteristics associated with the speech such that the speech is generated using a volume level, a pitch level, a speech rate level, any identified accent, and / or the like.

[0022] Then, the system can cause the character to output speech using at least the audio data. Thus, by performing one or more of the processes described herein, the speech output by the character can better convey the emotional state associated with the character. In some examples, when the character continues to communicate with the user and / or one or more other characters, the system can continue to perform these processes. For example, the system can continue to perform these processes to update the emotional state of the character for each letter, symbol, number, punctuation mark, word, sentence, paragraph, and / or the like associated with the speech output by the character.

[0023] The system can use one or more techniques to train a second machine learning model (and / or the combined first and second machine learning models) to generate emotion information, which is then used to convey emotion in speech. For example, the system can use prompt tuning, prompt engineering, and / or any other training techniques to train the second machine learning model. As described herein, during training, the second machine learning model can be trained to determine the emotional state associated with the speech, as well as to determine one or more values of one or more variables associated with the emotional state and / or the speech. For example, the second machine learning model can be trained using input data and corresponding ground truth data representing the values of the emotional state and / or the variables. Techniques for training one or more machine learning models are described in more detail herein.

[0024] The systems and methods described herein can be used for but are not limited to non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, robots or robotic platforms with and without drivers, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, submarines, drones, and / or other vehicle types. Additionally, the systems and methods described herein can be used for various purposes, such as but not limited to, for machine control, machine movement, machine drive, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transmission simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0025] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, sensing systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, medical systems, boating systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulation, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0026] Referring to Figure 1A , Figure 1A , a first example data flow diagram of a first process 100 for generating speech expressing an emotional state using one or more machine learning models in accordance with some embodiments of the present disclosure is shown. It should be understood that such and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or combined with other components and implemented in any suitable combination and location. The different functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the different functions may be performed by a processor executing instructions stored in a memory.

[0027] As shown, process 100 may include: text component 102 receiving input data 104. As shown, input data 104 may include user data 106, role data 108, and / or any other type of data that may be applied to text component 102. As described herein, user data 106 may include, but is not limited to, text data representing text input by one or more users (and / or text data generated from speech, such as through one or more translation, automatic speech recognition (ASR), diarization, and / or other speech-to-text (STT) processing models or algorithms), audio data representing user speech from the user, image data representing one or more images (e.g., videos) depicting the user and / or the user's environment, profile data representing information associated with the user (e.g., location, age, interests, personality traits, etc.), mood data representing one or more moods associated with the user, and / or any other type of data representing information associated with the user. In some examples, text component 102 may receive user data 106 based at least on user-provided consent. For example, the user may use one or more user devices to provide the information represented by user data 106.

[0028] Role data 108 may represent information associated with the role for which speech is to be output. As described herein, the information may include, but is not limited to, characteristics associated with the role (e.g., occupation, relationship, personality traits, etc.), past communications (e.g., past speech output by the role, etc.), current environment (e.g., current interaction with other roles, current location, current goals, etc.), and / or any other information associated with the role. For a first example, role data 108 may at least represent the current environment associated with the role, such as other roles with which the role is communicating, whether the role is friendly to other roles, the location of the role, etc. For a second example, role data 108 may represent received past text, past text (or speech) output, past mood states, and / or any other information associated with past communications associated with the role.

[0029] Then, process 100 may include: text component 102 processes at least a portion of input data 104 and generates and / or outputs text data 110 representing text, at least based on the processing. As described herein, text may include, but is not limited to, one or more letters, words, symbols, numbers, characters, punctuation marks, tags, and / or the like. In some examples, text component 102 may include and / or use one or more machine learning models (e.g., one or more large language models), one or more neural networks, one or more algorithms, one or more tables, and / or any other services, tools, and / or techniques to perform one or more of the processes described herein with respect to text component 102. For example, text component 102 may include one or more machine learning models that are trained to process input data 104 to generate text data 110, where the training is described in more detail herein.

[0030] In some examples, text data 110 may be generated in a format that can be processed later by one or more other components and / or models. For example, text data 110 may represent one or more tokens that represent text. In such examples, individual ones of these tokens may represent a portion of the text, such as a letter, word, symbol, number, character, punctuation mark, tag, and / or the like.

[0031] In some examples, text data 110 may represent a response to a user. For example, if user data 106 represents text associated with a comment, query, request, and / or the like, the text represented by text data 110 may include a response to the comment, query, and request, and / or the like. In some examples, text data 110 may represent text associated with a character communicating with one or more other characters. For example, if a character is communicating with other characters, the text may include words associated with the speech that the character is to output to the other characters.

[0032] For example, Figure 2 illustrates an example of generating text associated with a user input according to some embodiments of the present disclosure. As Figure 2As shown in the example of, the text component 102 can receive input data 202 that represents text input by a user (e.g., via speech, an input device, etc.), which can represent and / or include input data 104, where the text includes the words "How are you doing today?" Then, the text component 102 can be configured to process the input data 202 (e.g., using one or more machine learning models), and at least based on this processing, generate text data 204 that represents additional text associated with a response. For example, and as shown, the text can include the words "I am doing great, it is nice to see you." Although Figure 2 the example of only shows the input data 202 as including text, in other examples, the input data 202 can include any other type of input data described herein.

[0033] In Figure 2 the example of, the text data 204 can represent a series of tokens associated with the text. For example, the text data 204 can represent one or more first tokens for the word "I", one or more second tokens for the word "am", one or more third tokens for the word "great", and so on. In other examples, the text can be tokenized in any suitable way to be processed using one or more machine learning models (e.g., an LLM). In some examples, the text component 102 can generate text data 204 that represents these tokens so that additional components and / or models can process the text data 204.

[0034] Returning to the example of Figure 1A the process 100 can include: the sentiment component 112 receiving at least a portion of the input data 104 and / or at least a portion of the text data 110. Then, the process 100 can include: the sentiment component 112 processing at least this portion of the input data 104 and / or at least this portion of the text data 110, and at least based on this processing, generating and / or outputting sentiment data 114 associated with the text. In some examples, the sentiment component 112 can include and / or use one or more machine learning models (e.g., one or more large language models), one or more neural networks, one or more algorithms, one or more tables, and / or any other service, tool, and / or technology to perform one or more of the processes described herein with respect to the sentiment component 112. Additionally, the sentiment data 114 can represent sentiment information, such as at least a sentiment state and one or more values of one or more variables associated with the sentiment state and / or the speech.

[0035] As described herein, emotional states can include, but are not limited to, anger, calm, disgust, fear, happiness, helpfulness, humor, sadness, and / or any other emotional state. Additionally, variables associated with an emotional state can at least include the intensity of the emotional state. Thus, a value can indicate an intensity level, such as very low, low, medium, high, very high, and / or any other intensity level associated with the emotional state. Further, variables associated with speech can include, but are not limited to, volume, pitch, resonance, clarity, speech rate, accent, and / or any other characteristic associated with speech. Thus, a value associated with such a variable can indicate one or more levels and / or degrees associated with the variable. For a first example, values for volume can include muted, ultra-low, low, medium, high, ultra-high, and / or the like. For a second example, values associated with pitch and / or speech rate can include ultra-low, low, medium, high, ultra-high, and / or the like.

[0036] In some examples, the emotional data 114 can represent values using any technique. For a first example, for an emotional state, the value can include a first marker and / or label for anger, a second marker and / or label for calm, a third marker and / or label for happiness, a fourth marker and / or label for helpfulness, and so on. For a second example, also for an emotional state, the value can include a first value and / or string for anger (e.g., 1), a second value and / or string for calm (e.g., 2), a third value and / or string for happiness (e.g., 3), a fourth value and / or string for helpfulness (e.g., 4), and so on. For a third example, also for an emotional state, each type of emotional state can be associated with one or more bytes associated with the emotional data. Thus, the byte associated with the determined emotional state can include one or more first values (e.g., 1) and the byte associated with other emotional states can include one or more second values (e.g., 0). Additionally, similar techniques can be used for values associated with intensity and / or characteristics.

[0037] Although Figure 1AThe example of illustrates that the emotion data 114 is separated from the text data 110, but in some examples, the emotion data 114 may include at least part of the text data 110. For example, the emotion component 112 may generate the emotion data 114 by adding emotion information to the text data 110. For example, if the text data 110 represents a series of tokens associated with the text (e.g., a character's response), the emotion component 112 may generate the emotion data 114 by adding tags associated with the emotion state and the values of the variables to the text data. In such an example, the tags associated with a single emotion state may be associated with one or more tokens. For example, the first set of tags associated with the first determined emotion state may be associated with the first set of tokens, the second set of tags associated with the second determined emotion state may be associated with the second set of tokens, the third set of tags associated with the third determined emotion state may be associated with the third set of tokens, and so on.

[0038] For example, Figure 3 illustrates an example of determining emotion information associated with the speech output by a character according to some embodiments of the present disclosure. As Figure 3 shown in the example of , the emotion component 112 may receive the input data 202 and / or the text data 204. Then, the emotion component 112 may be configured to process the input data 202 and the text data 204, and at least based on this processing, generate the emotion data 302 representing the emotion information associated with the text (which may represent and / or include the emotion data 114). For example, the emotion data 302 may represent the value 304 associated with the emotion state 306 and the value 308 associated with the intensity 310 of the emotion state 306. As described herein, the value 304 may indicate anger, calm, disgust, fear, happiness, helpfulness, humor, sadness, and / or any other emotion state. In addition, the value 308 may indicate a very low, low, medium, high, very high, and / or any other intensity level associated with the emotion state.

[0039] The emotion data 302 further represents values 312(1)-(4) of different characteristics 314(1)-(4) of speech (also referred to singly as "characteristic 314" or plurally as "characteristics 314"). For example, if the first characteristic 314(1) includes volume, the first value 312(1) may indicate mute, ultra-low, low, medium, high, ultra-high, and / or the like. Additionally, if the second characteristic 314(2) includes pitch, the second value 312(2) may indicate ultra-low, low, medium, high, ultra-high, and / or the like. Additionally, if the third characteristic 314(3) includes speech rate, the third value 312(3) may indicate ultra-low, low, medium, high, ultra-high, and / or the like. Additionally, if the fourth characteristic 314(4) indicates stress on at least a portion of the text, the fourth value 312(4) may indicate a first value (e.g., 0) if that at least a portion of the text should not be stressed, or the fourth value 312(4) may indicate a second value (e.g., 1) if that at least a portion of the text should be stressed.

[0040] Although Figure 3 the example of shows the emotion data 302 as being separate from the text data 204, in other examples, the emotion data 302 may include at least a portion of the text data 204. For example, the emotion component 112 may generate the emotion data 302 by adding emotion information to the text data 204. In such an example, the emotion data 302 may thus represent a series of markers associated with the text from the text data 204 and labels associated with the emotion information. For example, the emotion data 302 may represent one or more first labels associated with the value 304 of the emotion state 306, one or more second labels associated with the value 308 of the intensity 310, one or more third labels associated with the first value 312(1) of the first characteristic 314(1), one or more fourth labels associated with the second value 312(2) of the second characteristic 314(2), one or more fifth labels associated with the third value 312(3) of the third characteristic 314(3), and / or one or more sixth labels associated with the fourth value 312(4) of the fourth characteristic 314(4).

[0041] As Figure 1AAs further shown in the example of, in some examples, the input data 104 applied to the emotion component 112 may include cue data 116 representing one or more cues (e.g., one or more tokens) for causing the emotion component 112 to generate a particular type of emotion information. For a first example, if the emotion component 112 generates emotion data 114 including a particular value of an emotional state (such as a value associated with anger), the cue data 116 may represent one or more cues (e.g., one or more tokens) that cause the emotion component 112 to generate that value of the emotional state. For a second example, if the emotion component 112 generates emotion data 114 including a particular value of the intensity of an emotional state (such as a value associated with medium), the cue data 116 may represent one or more cues (e.g., one or more tokens) that cause the emotion component 112 to generate that value of the intensity. Also, for a third example, if the emotion component 112 generates emotion data 114 including a particular value of a characteristic of speech (such as a value associated with medium speech rate), the cue data 116 may represent one or more cues (e.g., one or more tokens) that cause the emotion component 112 to generate that value of the characteristic. In examples where cue data 116 is used, the cue data 116 may be learned during the training of the emotion component 112.

[0042] Return reference Figure 1A In an example of, the process 100 may include: the speech component 118 receives at least a portion of the text data 110 and / or at least a portion of the emotion data 114. Then, the process 100 may include: the speech component 118 processes at least a portion of the text data 110 and / or at least a portion of the emotion data 114, and based at least on the processing, generates audio data 120 representing speech. In some examples, the speech component 118 may include and / or use one or more machine learning models, one or more neural networks, one or more algorithms, one or more tables, and / or any other services, tools, and / or technologies to perform one or more of the processes described herein with respect to the speech component 118. For example, the speech component 118 may include a text-to-speech (TTS) service and / or model.

[0043] As described herein, the speech represented by the audio data 120 can be associated with the text represented by the text data 110 (e.g., including the words of the text). Additionally, the speech can be expressed at least based on the emotion information represented by the emotion data 114. For example, the audio data 120 can cause the speech to be spoken using the emotional state and / or the intensity of the emotional state represented by the emotion data 114. Additionally, the audio data 120 can cause the speech to be spoken using the value of a characteristic associated with the speech, such as a volume level, a pitch level, a speech rate, an accent when needed, and / or the like represented by the emotion data 114. In other words, the speech component 118 can be configured to generate audio data such that the character outputs the speech in an emotion-expressing manner.

[0044] For example, Figure 4 illustrates an example of generating emotion-expressing speech according to some embodiments of the present disclosure. As Figure 4 shown in the example, the speech component 118 can receive at least the text data 204 and the emotion data 302. Then, the speech component 118 can process the text data 204 and the emotion data 302, and based at least on this processing, generate audio data 402 representing the speech (which can represent and / or include the audio data 120). As shown, the speech includes the text "I'm doing well and very glad to see you today." Then, the audio data 402 can be used to cause the character 404 to output the speech, which can be represented by 406. For example, the character 404 can output the speech in a manner that accentuates the intensity of the emotional state 304 and / or the characteristic 314 of the speech 406.

[0045] For example, the speech 406 output by the character 404 can be associated with an intensity level indicated by the value 308 of the intensity 310. Additionally, the volume of the speech 406 can be based on the volume level indicated by the first value 312(1), the pitch in the speech 406 can be based on the pitch level indicated by the second value 312(2), the speech rate of the speech 406 can be based on the speech rate indicated by the third value 312(3), and one or more parts of the speech can be accentuated based on the fourth value 312(4).

[0046] Returning to the Figure 1A example, the process 100 can continue to repeat to generate additional audio data 120 representing additional speech output by the character. For example, the process 100 can repeat to generate the audio data 120 for each letter, symbol, number, punctuation mark, word, sentence, paragraph, and / or the like associated with the speech output by the character. In this way, when the character continues to communicate with the user and / or other characters, the emotional state of the character can continue to be updated.

[0047] As described herein, in some examples, by performing Figure 1AIn process 100, the audio data 120 can represent speech expressed using different emotional states, even for text data 110 that represents the same text. For example, at least based on processing the first input data 104, the text component 102 can generate first text data 110 representing the text. Then, the emotion component 112 can process the first input data 104 and / or the first text data 110, and at least based on this processing, generate first emotion data 114 representing first emotion information associated with the text. Additionally, at least based on processing second input data 104 that is at least partially different from the first input data 104, the text component 102 can generate second text data 110 representing the same text. However, then the emotion component 112 can process the second input data 104 and / or the second text data 110, and at least based on this processing, generate second emotion data 114 representing second emotion information associated with the text. For example, even though the first emotion information and the second emotion information are both associated with the same text, the emotional states, the intensities of the emotional states, and / or one or more values of one or more variables between them can be different. Thus, process 100 can generate speech for a character that better expresses the actual emotional state based on the circumstances surrounding these exchanges.

[0048] Although Figure 1A the example of shows an example layout of a speech system, according to sequential examples, the speech system can include different layouts. For example, Figure 1B shows a second example data flow diagram of a second process 122 for generating speech expressing an emotional state using one or more machine learning models according to some embodiments of the present disclosure. As shown, process 122 can include: a processing component 124 receives input data 126, where the input data 126 includes user data 128, character data 130, prompt data 132, and / or any other type of data. For example, in some examples, the input data 126, user data 128, character data 130, and / or prompt data 132 can be respectively similar to and / or include the user's input data 104, user data 106, character data 108, and / or prompt data 116.

[0049] Then, process 122 may include processing component 124 processing input data 126 and, at least based on the processing, generating and / or outputting data 134. In some examples, processing component 124 may include and / or use one or more machine learning models (e.g., one or more large language models (LLMs)), one or more neural networks, one or more algorithms, one or more tables, and / or any other services, tools, and / or technologies to perform one or more of the processes described herein with respect to processing component 124. As shown, processing component 124 may at least include text component 136 (which may be similar to and / or include text component 102) and sentiment component 138 (which may be similar to and / or include sentiment component 112).

[0050] For example, if processing component 124 includes one or more machine learning models, text component 136 may include one or more layers and / or one or more channels of a machine learning model trained to generate text data 140 and sentiment component 138 may include one or more layers and / or one or more channels of a machine learning model trained to generate sentiment data 142. In some examples, text data 140 and / or sentiment data 142 may be similar to and / or include text data 110 and / or sentiment data 114, respectively. In other words, processing component 124 may be trained to output data 134 that includes both text data 140 and sentiment data 142.

[0051] For example, Figure 5 illustrates an example of determining text and sentiment information associated with speech output by a character according to some embodiments of the present disclosure. As Figure 5 shown in the example of, processing component 124 may receive input data 202 (which may represent and / or include input data 126). Then, processing component 124 may process input data 202 and, at least based on the processing, generate output data 502 (which may represent and / or include output data 134), the output data including text data 204 (which may represent and / or include text data 140) and sentiment data 302 (which may represent and / or include sentiment data 142).

[0052] Returning to the reference Figure 1B of the example, process 122 may include: speech component 118 receiving at least a portion of output data 134. Then, process 122 may include: speech component 118 processing the at least a portion of output data 134 and, at least based on the processing, generating audio data 144 representing speech. In some examples, audio data 144 may represent and / or include audio data 120.

[0053] Figure 6A data flow diagram of a process 600 for training one or more models 602 to generate emotion information associated with speech is shown, in accordance with some embodiments of the present disclosure. In some examples, the model 602 may include an emotion component 112 and / or a processing component 124 and / or be used by the emotion component 112 and / or the processing component 124 (e.g., emotion component 142). As shown, the model 602 may be trained using input data 604. In some examples, the input data 604 may be similar to the input data 104 and / or the input data 126. For example, the input data 604 may include user data associated with one or more users and / or role data associated with one or more roles. In some examples, such as in the example where the model 602 is separate from the text component 102, as shown in Figure 1A the example shown, the input data 604 may further include text data representing text. For example, the input data 126 may represent text data 110 generated using the text component 102.

[0054] The model 602 may be trained using the training input data 604 and corresponding ground truth data 606. The ground truth data 606 may include annotations, labels, masks, and / or the like. For example, and as shown, the ground truth data 606 may represent values associated with different emotions and / or speech, such as emotion state values 608 indicating different emotion states that the model 602 is trained to detect, intensity values 610 indicating different intensity levels that the model 602 is trained to detect, and / or characteristic values 612 indicating different speech characteristic levels that the model 602 is trained to detect. The ground truth data 606 may be synthetically generated (e.g., from a computer model or rendering), real-world generated (e.g., designed and generated from real-world data), machine automated (e.g., using feature analysis and learning to extract features from data and then generate labels), manually annotated (e.g., by an annotator or annotation expert defining the location of the labels), and / or combinations thereof. In some examples, for each instance of the input data 604, there may be a corresponding ground truth data 606.

[0055] As Figure 6As further shown, the training engine 614 can use one or more loss functions that measure the loss (e.g., error) in the output 616 compared to the ground truth data 606. In some examples, the output 616 can be similar to the sentiment data 114 and / or the sentiment data 142. For example, the output 616 can indicate a value of an emotional state, a value of an intensity, and / or a value of a speech characteristic. Any type of loss function can be used, such as cross-entropy loss, mean squared error, mean absolute error, mean bias error, and / or other loss function types. In some examples, different outputs 616 can have different loss functions. For example, the emotional state value can have a first loss function, the intensity value can have a second loss function, and / or one or more characteristic values (e.g., values associated with each type of characteristic) can have a corresponding third loss function. In these examples, the loss functions can be combined to form a total loss, and the total loss can be used to train the model 602 (e.g., update the parameters of the model 602). In any example, a backpropagation calculation can be performed to recursively calculate the gradient of the loss function with respect to the training parameters. In some examples, the weights and biases of the model 602 can be used to calculate these gradients.

[0056] In some examples, one or more additional techniques can be used to train the model 602, such as to improve the efficiency of training. For example, the model 602 can be trained to determine different variables in different training instances. For example, during a first training instance, the model 602 can be trained to determine a value associated with the emotional state of speech. Additionally, during a second training instance, the model 602 can be trained to determine a value associated with the intensity of the emotional state. Additionally, during a third training instance, the model 602 can be trained to determine a value of a first characteristic of the speech. Then, the technique can continue to train the model to determine values of one or more other variables associated with determining emotional information.

[0057] Furthermore, one or more techniques (e.g., p-tuning, prompt-tuning, LoRA, prompt engineering, etc.) can be used to determine one or more prompts associated with causing the model 602 to generate specific emotional information, where these prompts can be represented by the prompt data 116 and / or the prompt data 132. For example, the training can include determining one or more prompts that cause the model 602 to determine one or more values of one or more specific emotional states, one or more prompts that cause the model 602 to determine one or more values of one or more intensity levels, and / or one or more prompts that cause the model 602 to determine one or more values of one or more characteristic levels associated with the speech. In some examples, the process of determining the prompts can be complementary to, or as an alternative to, the process of updating the model 602 (e.g., updating the parameters of the model 602) during training.

[0058] Now refer to Figure 7 and Figure 8 , each block of methods 700 and 800 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, different functions can be performed by a processor executing instructions stored in a memory. Methods 700 and 800 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 700 and 800 can be provided by a stand-alone application, service, or hosted service (individually or in combination with another hosted service) or a plug-in for another product, to name a few. Additionally, by way of example, methods 700 and 800 are described with respect to Figure 1A - Figure 1B . However, these methods 700 and 800 can additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein.

[0059] Figure 7 FIG. shows a flowchart of a method 700 for enabling a character to communicate using speech expressing emotions, according to some embodiments of the present disclosure. At block B702, the method 700 can include: generating second data representing an emotional state associated with the text and one or more variables associated with at least one of the emotional state or the speech corresponding to the text, using one or more machine learning models and based at least on first data representing one or more inputs. For example, an emotion component 112 (e.g., one or more machine learning models) can process input data 104 and / or text data 110 (e.g., the first data). Based at least on this processing, the emotion component 112 can generate emotion data 114 (e.g., the second data) representing the emotional state and one or more variables. As described herein, the emotion data 114 can represent at least the value of the emotional state and the corresponding values of one or more (e.g., each) of the variables.

[0060] Although this example describes the emotion component 112 processing the input data 104 and / or text data 110 to generate the emotion data 114, in other examples, a processing component 124 (e.g., a machine learning model) can process input data 126 (e.g., the first data) to generate output data 134 (e.g., the second data). As described herein, the output data 134 can include text data 140 and emotion data 142.

[0061] At block B704, the method 700 may include: generating audio data representing speech and at least based on an emotional state, at least based on second data. For example, the speech component 118 may process the emotion data 114 and / or the text data 110 (and / or the output data 134). At least based on this processing, the speech component 118 may generate audio data 120 (and / or audio data 144) representing speech expressed using the emotional state. For example, the audio data 120 may cause the speech to be spoken using the emotional state represented by the emotion data 114 and / or the intensity of the emotional state. Additionally, the audio data 120 may cause the speech to be spoken using values of characteristics associated with the speech, such as volume level, pitch level, speech rate, stress when needed, and / or the like represented by the emotion data 114.

[0062] At block B706, the method 700 may include: animating a character using at least the speech. For example, the audio data 120 may be used to animate the character, where the animation includes the character outputting speech in a manner that expresses the emotional state.

[0063] Figure 8 A flowchart of a method 800 for generating audio data representing speech expressing an emotion is shown, in accordance with some embodiments of the present disclosure. At block B802, the method 800 may include: generating second data representing text, using first data representing one or more inputs. For example, the text component 102 may receive input data 104 (e.g., the first data) representing one or more inputs. As described herein, the input data 104 may include user data 106 and / or character data 108. Then, the text component 102 may process the input data 104, and at least based on this processing, generate text data 110 (e.g., the second data) representing text.

[0064] At block B804, the method 800 may include: generating third data representing an emotional state associated with the text and one or more variables associated with at least one of the emotional state or the speech corresponding to the text, using one or more machine learning models and at least based on the second data. For example, the emotion component 112 (e.g., a machine learning model) may process the text data 110. In some examples, the emotion component 112 may further process the input data 104. At least based on this processing, the emotion component 112 may generate emotion data 114 (e.g., the third data) representing the emotional state and one or more variables. As described herein, the emotion data 114 may represent at least the value of the emotional state and the corresponding values of one or more (e.g., each) of the variables.

[0065] At block B806, method 800 may include: generating audio data that represents speech and is expressed using an emotional state, based at least on second data and third data. For example, speech component 118 may process emotion data 114 and / or text data 110. Based at least on this processing, speech component 118 may generate audio data 120 that represents the speech expressed using the emotional state. For example, audio data 120 may cause the speech to be spoken using the emotional state represented by emotion data 114 and / or the intensity of the emotional state. Additionally, audio data 120 may cause the speech to be spoken using values of characteristics associated with the speech, such as volume level, pitch level, speech rate, accent when needed, and / or the like represented by emotion data 114.

[0066] Example computing device

[0067] Figure 9 is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. Computing device 900 may include an interconnect system 902 that directly or indirectly couples the following devices: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays), and one or more logic units 920. In at least one embodiment, one or more computing devices 900 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Accordingly, one or more computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of a GPU dedicated to computing device 900), or a combination thereof.

[0068] Although Figure 9 the various blocks are shown as being connected by lines via interconnect system 902, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 918 (such as a display device) may be considered an I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device in addition to the memory of GPU 908, CPU 906, and / or other components). In other words, Figure 9The computing devices described are merely illustrative. No distinction is made among such categories as "workstations," "servers," "laptop computers," "desktop computers," "tablet computers," "client devices," "mobile devices," "handheld devices," "gaming consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are considered within the scope of Figure 9 the computing devices.

[0069] The interconnect system 902 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 can be directly connected to the memory 904. Further, the CPU 906 can be directly connected to the GPU 908. In cases where there are direct or point-to-point connections between components, the interconnect system 902 can include a PCIe link to effect the connection. In these examples, the PCI bus need not be included in the computing device 900.

[0070] The memory 904 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 900. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0071] Computer storage media can include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, the memory 904 can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic tape cartridges, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 900. As used herein, computer storage media does not include signals per se.

[0072] A computer storage medium can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information delivery medium. The term "modulated data signal" can refer to a signal that sets or changes one or more of its characteristics in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as a wired network or direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the foregoing should also be included within the scope of computer-readable media.

[0073] One or more CPUs 906 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more CPUs 906 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of simultaneously processing numerous software threads. One or more CPUs 906 can include any type of processor, and depending on the type of computing device 900 implemented, can include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor can be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary co-processors (such as a math co-processor), computing device 900 can include one or more CPUs 906.

[0074] In addition to or instead of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 908 may be integrated GPUs (e.g., where one or more of the CPUs 906 and / or one or more of the GPUs 908 may be discrete GPUs). In an embodiment, one or more of the one or more GPUs 908 may be a coprocessor of one or more of the one or more CPUs 906. One or more GPUs 908 may be used by the computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 908 may be used for general-purpose computing on the GPU (GPGPU). One or more GPUs 908 may include hundreds or thousands of cores capable of simultaneously processing hundreds or thousands of software threads. One or more GPUs 908 may generate pixel data for an output image in response to a rendering command (e.g., a rendering command received from one or more CPUs 906 via a host interface). One or more GPUs 908 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 904. One or more GPUs 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 908 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0075] In addition to or alternatively to one or more CPUs 906 and / or one or more GPUs 908, one or more logic units 920 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. In an embodiment, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more of the logic units 920 may be one or more of the one or more CPUs 906 and / or one or more of the one or more GPUs 908 and / or integrated in one or more of the one or more CPUs 906 and / or one or more of the one or more GPUs 908, and / or one or more of the logic units 920 may be discrete components or otherwise external to one or more of the one or more CPUs 906 and / or one or more of the one or more GPUs 908. In an embodiment, one or more of the logic units 920 may be a coprocessor of one or more of the one or more CPUs in one or more of the one or more CPUs 906 and / or one or more of the one or more GPUs in one or more of the one or more GPUs 908.

[0076] Examples of one or more logic units 920 include one or more processing cores and / or their components, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel vision cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0077] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 900 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 910 may include components and functionality for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly send data received via the network and / or through the interconnect system 902 to one or more GPUs 908 (e.g., the memory of one or more GPUs 908).

[0078] The I / O port 912 may enable the computing device 900 to be logically coupled to other devices including I / O components 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated in) the computing device 900. Illustrative I / O components 914 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. The I / O components 914 may provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, inputs may be sent to appropriate network elements for further processing. The NUI may implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of the computing device 900. The computing device 900 may include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 900 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables the detection of motion. In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0079] The power supply 916 may include hard-wired power, battery power, or a combination thereof. The power supply 916 may provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0080] One or more presentation components 918 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The one or more presentation components 918 may receive data from other components (e.g., one or more GPUs 908, one or more CPUs 906, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0081] Example data center

[0082] Figure 10 An example data center 1000 that may be used in at least one embodiment of the present disclosure is shown. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0083] As Figure 10 shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources (“node C.R.”) 1016(1)-1016(N), where “N” represents any whole positive integer. In at least one embodiment, the node C.R. 1016(1)-1016(N) may include, but is not limited to, any number of central processing units (“CPU”) or other processors (including DPU, accelerators, field programmable gate arrays (FPGA), graphics processors or graphics processing units (GPU), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VM”), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R. 1016(1)-1016(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, the node C.R. 1016(1)-10161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of the node C.R. 1016(1)-1016(N) may correspond to a virtual machine (VM).

[0084] In at least one embodiment, the grouped computing resources 1014 may include separate groupings of node C.R.s 1016 housed within one or more racks (not shown), or separate groupings of many racks within data centers located at different geographical locations (also not shown). Separate groupings of node C.R.s 1016 within the grouped computing resources 1014 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1016 including CPUs, GPUs, DPUs, and / or other processors may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0085] The resource coordinator 1012 may configure or otherwise control one or more node C.R.s 1016(1)-1016(N) and / or the grouped computing resources 1014. In at least one embodiment, the resource coordinator 1012 may include a software design infrastructure (“SDI”) management entity for the data center 1000. The resource coordinator 1012 may include hardware, software, or some combination thereof.

[0086] In at least one embodiment, as Figure 10As shown, the framework layer 1020 may include a job scheduler 1028, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. The framework layer 1020 may include a framework for software 1032 of the support software layer 1030 and / or one or more applications 1042 of the application layer 1040. The software 1032 or the application 1042 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1020 may be, but is not limited to, a type of free and open-source software web application framework that can perform large-scale data processing (e.g., "big data") using the distributed file system 1038, such as ApacheSparkTM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1028 may include a Spark driver to facilitate scheduling of the workloads supported by the various layers of the data center 1000. The configuration manager 1034 may be capable of configuring different layers, such as the software layer 1030 including Spark and the distributed file system 1038 and the framework layer 1020, to support large-scale data processing. The resource manager 1036 may be capable of managing the computing resources mapped to the distributed file system 1038 and the job scheduler 1028 or allocated to support the distributed file system 1038 and the job scheduler 1028 in a cluster or group. In at least one embodiment, the computing resources in the cluster or group may include the grouped computing resources 1014 at the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources.

[0087] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used by at least a portion of the nodes C.R. 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0088] In at least one embodiment, one or more applications 1042 included in the application layer 1040 may include one or more types of applications used by at least a portion of nodes C.R. 1016(1)-1016(N) of the framework layer 1020, grouped computing resources 1014, and / or distributed file system 1038. The one or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in connection with one or more embodiments.

[0089] In at least one embodiment, any one of the configuration manager 1034, resource manager 1036, and resource coordinator 1012 may implement any number and type of self-modifying actions based on any amount and type of data obtained in any technically feasible manner. The self-modifying actions may save the data center operator of the data center 1000 from making potentially poor configuration decisions and may avoid underutilization and / or poorly performing portions of the data center.

[0090] According to one or more embodiments described herein, the data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to the data center 1000. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center 1000 by using weight parameters calculated by one or more training techniques (such as, but not limited to, those described herein).

[0091] In at least one embodiment, the data center 1000 may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or their corresponding virtual computing resources) to perform training and / or inference using the above resources. In addition, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform a service for inferring information, such as image recognition, speech recognition, or other artificial intelligence services.

[0092] Example Network Environment

[0093] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of a computing device 900 as described in Figure 9 —e.g., each device may include similar components, features, and / or functionality of one or more computing devices 900. Additionally, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1000, an example of which is described in more detail herein with respect to Figure 10 more detail.

[0094] The components of the network environment may communicate with each other via one or more networks, which may be wired, wireless, or both. The network may include multiple networks or one network among multiple networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or the public switched telephone network (PSTN), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide a wireless connection.

[0095] A compatible network environment may include one or more peer-to-peer network environments (in which case, servers may not be included in the network environment) and one or more client-server network environments (in which case, one or more servers may be included in the network environment). In a peer-to-peer network environment, the functions described herein with respect to one or more servers may be implemented on any number of client devices.

[0096] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework that supports software layers and / or one or more applications of an application layer. The software or application may respectively include network-based service software or applications. In an embodiment, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework, such as may be used for large-scale data processing (e.g., “big data”) using a distributed file system.

[0097] Cloud-based network environments can provide cloud computing and / or cloud storage that perform any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across a state, region, country, the globe, etc.). If the connection to a user (e.g., a client device) is relatively close to one or more edge servers, the central or core server(s) can assign at least a portion of the function to one or more edge servers. Cloud-based network environments can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0098] One or more client devices can include at least some of the components, features, and functions of one or more of the example computing devices 900 described herein. By way of example, and not limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, spacecraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device. Figure 9

[0099] The present disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0100] ​As used herein, the recitation of "and / or" between two or more elements should be construed to mean only a combination of one element or more. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Further, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Still further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0101] The subject matter of the present disclosure is specifically described herein to meet statutory requirements. However, this description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may be embodied in other ways, in combination with other current or future technologies, to include different steps or combinations of steps similar to those described in this document. Additionally, although the terms "step" and / or "block" may be used herein to imply different elements of a method employed, these terms should not be construed to imply any particular order among or between the various steps disclosed herein, unless the order of the individual steps is explicitly described.

[0102] Example paragraphs

[0103] A: A method, comprising: using one or more language models and generating second data representing an emotional state associated with text and one or more variables associated with at least one of the emotional state or speech associated with the text, based at least on first data representing one or more inputs; generating audio data representing the speech based at least on the second data, based at least on the emotional state and the one or more variables; and animating a character using at least the speech.

[0104] B: The method according to paragraph A, further comprising at least one of: using one or more language models and generating third data representing the text, based at least on the first data; or using one or more second language models to generate third data representing the text.

[0105] C: The method according to paragraph A or paragraph B, wherein: the one or more variables include at least an intensity associated with the emotional state; and the second data further represents a value associated with the intensity.

[0106] D: A method according to any one of paragraphs A - C, wherein: one or more variables include one or more characteristics associated with the speech, the one or more characteristics including at least one of volume, speech rate, pitch, or stress associated with the speech; and the second data further represents one or more values associated with the one or more characteristics.

[0107] E: A method according to any one of paragraphs A - D, wherein: the one or more variables at least include an intensity level associated with the emotional state and one or more characteristics associated with the speech; the second data further represents a first value associated with the intensity level and one or more second values associated with one or more levels of the one or more characteristics; and generating the audio data representing the speech includes generating the audio data at least based on the emotional state, the first value, and the one or more second values such that the speech expresses the emotional state using the intensity level and the one or more characteristic levels.

[0108] F: A method according to any one of paragraphs A - E, wherein: the first data includes at least one of the following: first input data associated with a user, the first input data including at least one of text data representing input text, second audio data representing the user's speech, or image data representing one or more images corresponding to the user; or second input data associated with a character, the second input data representing at least one of one or more characteristics associated with the character, one or more situations associated with the character, or one or more interactions or one or more past communications associated with the character.

[0109] G: A method according to any one of paragraphs A - F, wherein: the first data further represents one or more first values associated with the one or more variables; and the method further includes: using one or more language models and generating fourth data representing a second emotional state associated with the text and one or more second values associated with the one or more variables at least based on third data representing one or more second inputs.

[0110] H: A method according to any one of paragraphs A - G, wherein: the second data is associated with a first part of the text and further represents one or more first values of the one or more variables; and the method further comprises: using the one or more language models and generating third data associated with a second part of the text based at least on the first data, the third data representing a second emotional state and one or more second values associated with the one or more variables; generating second audio data representing a second speech associated with the second part of the text based at least on the third data, the second speech being based at least on the second emotional state and the one or more second values associated with the one or more variables; and animating a character using at least the second speech.

[0111] I: A method according to any one of paragraphs A - H, wherein: the text comprises one or more words; and the speech comprises the one or more words spoken using the emotional state and based at least on one or more variables.

[0112] J: A system comprising: one or more processing units for: generating first data representing a text based at least on input data; using one or more language models and generating second data representing an emotional state associated with the text and one or more variables associated with at least one of the emotional state or the speech associated with the text based at least on the first data; and generating audio data representing speech based at least on the emotional state based at least on the first data and the second data.

[0113] K: The system according to paragraph J, wherein at least one of the following: the generation of the text data uses one or more language models; or the generation of the text data uses one or more second language models.

[0114] L: The system according to paragraph J or paragraph K, wherein: the one or more variables at least comprise an intensity associated with the emotional state; and the second data further represents a value associated with the intensity.

[0115] M: The system according to any one of paragraphs J - L, wherein: the one or more variables comprise one or more characteristics associated with the speech, the one or more characteristics comprising at least one of volume, speech rate, pitch, or stress associated with the speech; and the second data further represents one or more values associated with the one or more characteristics.

[0116] N: A system according to any one of paragraphs J-M, wherein: the one or more variables include at least an intensity associated with the emotional state and one or more characteristics associated with the speech; the second data further represents a first value associated with an intensity level of the intensity and one or more second values associated with one or more characteristic levels of the one or more characteristics; and generating the audio data representing the speech includes: generating the audio data based at least on the emotional state, the first value, and the one or more second values such that the speech expresses the emotional state using the intensity level and the one or more characteristic levels.

[0117] O: A system according to any one of paragraphs J-N, wherein: the one or more processing units are further configured to: obtain input data associated with a user, the input data including at least one of second text data representing input text, second audio data representing the user's speech, or image data representing one or more images corresponding to the user, wherein the second data is further generated at least based on the input data.

[0118] P: A system according to any one of paragraphs J-O, wherein: the one or more processing units are further configured to: obtain input data associated with a role of the output speech, the input data representing at least one of one or more characteristics associated with the role, one or more situations associated with the role, one or more interactions associated with the role, or one or more past communications associated with the role, wherein the second data is further generated at least based on the input data.

[0119] Q: A system according to any one of paragraphs J-P, wherein the one or more processing units are further configured to animate the role at least based on the speech.

[0120] R: A system according to any one of paragraphs J-Q, wherein the system is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0121] S: A processor comprising: one or more processing units for generating audio data representing speech expressed using an emotional state, wherein the audio data is generated based at least on data representing an emotional state and one or more values associated with one or more variables associated with at least one of the emotional state or the speech, and the data representing the emotional state and the one or more values is determined using one or more large language models LLM.

[0122] T: A processor according to paragraph S, wherein the system is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: generating, using one or more language models and based at least on first data representing one or more inputs, second data representing an emotional state associated with a text and one or more variables associated with at least one of the emotional state or speech associated with the text; generating, based at least on the second data, audio data representing the speech based at least on the emotional state and the one or more variables; as well as The character is caused to be animated using at least the speech.

2. The method of claim 1, further comprising at least one of the following: generating third data representing the text using the one or more language models and based at least on the first data; or The third data representing the text is generated using one or more second language models.

3. The method of claim 1, wherein: The one or more variables include at least an intensity associated with the emotional state; and The second data further represents a value associated with the intensity.

4. The method of claim 1, wherein: The one or more variables include one or more characteristics associated with the speech, the one or more characteristics including at least one of volume, speech rate, pitch, or stress associated with the speech; and The second data further represents one or more values ​​associated with the one or more characteristics.

5. The method of claim 1, wherein: The one or more variables include at least an intensity level associated with the emotional state and one or more characteristics associated with the speech; The second data further represents a first value associated with the intensity level and one or more second values ​​associated with one or more levels of the one or more characteristics; as well as Generating the audio data representing the speech includes generating the audio data based at least on the emotional state, the first value, and the one or more second values ​​such that the speech expresses the emotional state using the intensity level and the one or more characteristic levels.

6. The method of claim 1, wherein the first data comprises at least one of the following: first input data associated with a user, the first input data comprising at least one of text data representing input text, second audio data representing user speech, or image data representing one or more images corresponding to the user; or Second input data associated with the character, the second input data representing at least one of one or more characteristics associated with the character, one or more situations associated with the character, one or more interactions associated with the character, or one or more past communications associated with the character.

7. The method of claim 1, wherein: The first data further represents one or more first values ​​associated with the one or more variables; and The method further includes generating, using the one or more language models and based at least on third data representing one or more second inputs, fourth data representing a second emotional state associated with the text and one or more second values ​​associated with the one or more variables.

8. The method of claim 1, wherein: The second data is associated with the first portion of the text and further represents one or more first values ​​of the one or more variables; as well as The method further comprises: generating, using the one or more language models and based at least on the first data, third data associated with a second portion of the text, the third data representing a second emotional state and one or more second values ​​associated with the one or more variables; generating, based at least on the third data, second audio data representing second speech associated with the second portion of the text, the second speech being based at least on the second emotional state and the one or more second values ​​associated with the one or more variables; as well as The character is caused to be animated using at least the second language.

9. The method of claim 1, wherein: The text includes one or more words; and The speech includes the one or more words spoken using the emotional state and based at least on the one or more variables.

10. A system comprising: One or more processing units for: generating first data representing text based at least on the input data; generating, using one or more language models and based at least on the first data, second data representing an emotional state associated with the text and one or more variables associated with at least one of the emotional state or speech associated with the text; as well as Based at least on the first data and the second data, audio data representing the speech based at least on the emotional state is generated.

11. The system of claim 10, wherein at least one of the following: The generation of the text data uses the one or more language models; or The text data is generated using one or more second language models.

12. The system of claim 10, wherein: The one or more variables include at least an intensity associated with the emotional state; and The second data further represents a value associated with the intensity.

13. The system of claim 10, wherein: The one or more variables include one or more characteristics associated with the speech, the one or more characteristics including at least one of volume, speech rate, pitch, or stress associated with the speech; and The second data further represents one or more values ​​associated with the one or more characteristics.

14. The system of claim 10, wherein: The one or more variables include at least an intensity associated with the emotional state and one or more characteristics associated with the speech; The second data further represents a first value associated with an intensity level of the intensity and one or more second values ​​associated with one or more characteristic levels of the one or more characteristics; as well as Generating the audio data representing the speech includes generating the audio data based on at least the emotional state, the first value, and the one or more second values ​​such that the speech expresses the emotional state using the intensity level and the one or more characteristic levels.

15. The system of claim 10, wherein the one or more processing units are further configured to: obtaining the input data associated with the user, the input data comprising at least one of second text data representing input text, second audio data representing user speech, or image data representing one or more images corresponding to the user, The second data is further generated based on at least the input data.

16. The system of claim 10, wherein the one or more processing units are further configured to: obtaining the input data associated with the character outputting the speech, the input data representing at least one of one or more characteristics associated with the character, one or more situations associated with the character, one or more interactions associated with the character, or one or more past exchanges associated with the character, The second data is further generated based on at least the input data.

17. The system of claim 10, wherein the one or more processing units are further configured to cause a character to be animated based at least on the speech.

18. The system of claim 10, wherein the system is included in at least one of: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language models (LLMs); A system for performing one or more conversational AI operations; Systems for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A processor comprising: One or more processing units for generating audio data representing speech expressed using an emotional state, wherein the audio data is generated based at least on data representing the emotional state and one or more values ​​associated with one or more variables associated with at least one of the emotional state or the speech, the data representing the emotional state and the one or more values ​​being determined using one or more large language models (LLMs).

20. The processor of claim 19, wherein the system is included in at least one of: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language models (LLMs); A system for performing one or more conversational AI operations; Systems for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images