Mood and psychological state-aware interaction with multimodal large-scale language models

Abstract

A system is described that uses a large-scale language model (LLM) with multimodal input having a low latency (within 500 milliseconds) response time. This low latency generates responses that lead to a more engaging and empathetic user experience, with all calculations performed on consumer hardware. Thus, the LLM uses message sentiment index analysis and / or facial behavior and / or voice parameters to derive the user's emotional state and lead to a more engaging and empathetic user experience. This is done by reviewing (a) the user's mood state while providing input prompts to the LLM in the current turn, (b) the user's mood and psychological state while reacting to the LLM's response in the previous turn, (c) the sentiment index of the LLM's response in the previous turn, and (d) the desired user mood and psychological state determined by the system's empathetic objective.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of the following applications, which are hereby incorporated by reference in their entirety: - U.S. Provisional Patent Application No. 63 / 469,725, filed April 18, 2023

[0002] The present disclosure relates to improved techniques for improving the multimodal output of large - language models (LLMs) using expressive behaviors obtained from multimodal inputs and moods and mental states measured from linguistic cues.

Background Art

[0003] LLMs such as ChatGPT and Llama are language models that have the ability to achieve other natural language processing tasks such as general - purpose language generation and classification. LLMs acquire these abilities by learning statistical relationships from text documents during computationally intensive self - supervised learning and semi - supervised learning processes. LLMs are sometimes used for text generation, which is a form of generative artificial intelligence (AI), by receiving input text and repeatedly predicting the next token or word.

[0004] By using expressive behaviors obtained from multimodal inputs and moods measured from linguistic cues, it is desirable to improve the text output of large - language models (LLMs, e.g., ChatGPT, Llama, etc.) to enhance the user experience and engagement. This approach combines linguistic and paralinguistic components extracted from multimodal inputs to construct a mood - aware input prompt for the LLM.

Summary of the Invention

[0005] This specification describes a system that uses an LLM with multimodal input having a low latency (within 500 milliseconds) response time. This low latency is necessary for a more engaging and empathetic user experience where all computations are executed on consumer hardware. By doing so, the system infers the user's mood and mental state from facial expressions, voice characteristics, text sentiment indices, and physiological measurements acquired while the user is interacting with the LLM. Guided by the inferred mood and mental state, the system customizes the LLM input prompt to create a more engaging and empathetic dialogue experience by considering the following: (a) The user's mood state while providing the input prompt to the LLM in the current turn, (b) The user's mood and mental state (e.g., confusion) while reacting to the LLM's response in the previous turn, (c) The sentiment index of the LLM's response in the previous turn, and (d) The desired user mood and mental state determined by the empathetic goals of the system. **Brief Description of the Drawings**

[0006] The accompanying drawings, in which like reference numerals refer to the same or functionally similar elements, are incorporated herein and form a part of this specification, and further illustrate embodiments of the claimed invention and serve to explain the various principles and advantages of those embodiments.

[0007] [Figure 1] FIG. 1 shows a schematic diagram of a system diagram of mood and mental state recognition-based interaction with a multimodal LLM.

[0008] [Figure 2] FIG. shows a schematic diagram of a state diagram of mood and mental state recognition-based interaction with a multimodal LLM.

[0009] [Figure 3]Figure 3 shows a schematic diagram of the computational flow chart for mood and psychological state-aware interaction with a multimodal LLM.

[0010] Those skilled in the art will understand that elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the drawings may be exaggerated relative to others in order to improve the understanding of embodiments of the invention.

[0011] Components of the apparatus and methods are represented in the drawings, where appropriate, by conventional symbols indicating only specific details relevant to understanding embodiments of the invention, so as not to obscure the disclosure with details that would be readily apparent to those skilled in the art who have a benefit of the description herein. [Modes for carrying out the invention]

[0012] I. VAD Model and its Use in LLM

[0013] A.VAD Model

[0014] A common model used in the field of emotional computing is the Valence-Arousal-Superiority (VAD) model (also known as the Pleasure-Arousal-Superiority (PAD) model). In these models, the X, Y, and Z axes range from -1 to 1 and are defined as follows:

[0015] Affective Valence - An index indicating the degree to which a subject feels pleasure or displeasure, ranging from -1 to 1. Here, -1 means the subject feels very negative / unpleasant, and 1 means the subject feels very positive / comfortable.

[0016] Dominance - Indicates the degree to which the subject has control over their emotions, on a scale from -1 to 1. Here, -1 means the subject has no control over their emotions, and 1 means the subject has complete control over their emotions.

[0017] Arousal Level - Indicates the degree to which the subject is awake or active, on a scale from -1 to 1. Here, -1 means the subject is very calm or very sleepy, and 1 means the subject is very active and energetic.

[0018] One of the fundamental concepts behind the model is that emotions can be defined by different combinations of these three dimensions.

[0019] B.LLMs

[0020] This disclosure defines “mood” as a temporal combination of an LLM user’s emotional (e.g., excitement, frustration, disappointment) and cognitive states (e.g., confusion, drowsiness).

[0021] Empathic Mood-Response LLM has an empathic objective to achieve a change in the user's mood. This objective is to maintain the user's mood, achieve higher valence, higher arousal, or lower arousal (all relative to the user's current mood), or guide the user to a specific area within the valence-arousal-superiority space. Based on the objective of the Empathic Mood-Response LLM, a target user mood is set within the VAD space, and the output expected to guide the user to that mood is calculated.

[0022] Empathic mood-responsive LLMs need to respond quickly in less than 500 milliseconds (human response latency is average 200 milliseconds), and this 500-millisecond response latency has been experimentally proven to be acceptable. This may not be achievable with multimodal LLMs that currently accept video and audio as inputs because video processing is computationally expensive and introduces latency. The objective of this disclosure is to provide low-latency response (<500 milliseconds).

[0023] Improved output can change the user's perceived mood. u and goal mood m tThe distance d to m is measured from the perspective of m , and the target mood is the apparent user mood m u and the empathic goal setting s of the LLM e is a function of: d m = m t - m u , m t = f(m u , s e ).

[0024] The LLM outputs a response y that emotionally aligns with its empathic goal. The mood value of the LLM output is m r equal to, and the empathic goal and user mood m t = f(m u , s e ) should be as close as possible. The empathic goal can be to mimic the user mood, improve the user mood, calm the user mood, etc. All empathic goals s e change the way f operates with respect to m u .

[0025] To facilitate mood - corresponding interactions with the LLM, a prompt customization module is developed. This module obtains estimates of the user's mood and mental state from face, voice, text, and physiological signals to calculate m u and the empathic goal s e and returns a customized text prompt y = g(m t ).

[0026] The LLM can output text, voice, or an animated anthropomorphic face with voice.

[0027] The LLM response mood value m r is estimated using a multimodal mood fusion module.

[0028] To achieve user mood across multiple responses, the empathetic mood-responsive LLM needs to know the mood of the most recent n conversation turns (where n is a small number) and compare it to the target mood. The mood-responsive LLM takes the user mood of the most recent n turns, the LLM output mood, and the user target mood as input and creates a mood-responsive LLM output for the next agent conversation turn.

[0029] User mood in the most recent n conversation turns (m u,i ) and LLM response mood (m r,i ) The state is, [(m u,i ,m r,i This is stored as i=-n:0, where i=0 is the current turn index.

[0030] Information from all modalities may be fused using transformation context recognition fusion technology to improve user emotional state recognition.

[0031] This customized prompt drives the LLM's response, which may be improved by injecting emotional information into the prompt, allowing the LLM to adjust its response.

[0032] LLM may run on the user's computing device or as a cloud service.

[0033] User video and audio may be captured using standard cameras and microphones on the computing device while interacting with the system.

[0034] Physiological signals may be captured by non-contact technologies such as photoplethysmography (PPG) and pupillary measurement via cameras, or by skin-contact sensors via wearable devices such as smartwatches, smart rings, and other sensors.

[0035] II. Diagram of mood and psychological state-awareness-based interaction with a multimodal large-scale language model.

[0036] Referring to Figure 1, a schematic diagram 100 of the system diagram for mood and psychological state recognition interaction with a multimodal LLM is shown. The system has inputs related to text from keyboard input 105, voice input 108, visual input 109, and biometric input 112, all from user 102. Text from keyboard input 105 is sent to the text input module 115, along with the voice-to-text conversion module 110 for voice input 108. The output of the text input module is sent to the language VAD module 120. Voice input 108 is sent to the voice VAD module 122, which is then sent to the psychological state estimator module 128. Visual input 109 is sent to the face VAD module 124, which is then sent to the psychological state estimator module 128. Biometric input 112 is sent to the wearable VAD module 126. Biometric input 112 may be obtained from any wearable device on the user 102's body, including headsets, armbands, legbands, etc.

[0037] The outputs from the VAD module 120 (language), VAD module 122 (voice), VAD module 124 (face), and VAD module 126 (wearable) are sent to the multimodal mood fusion module 130. The output of the multimodal mood fusion module 130, along with the output of the psychological state estimator module 128 and the text input module 115, is sent to the mood-sensitive LLM 132. Finally, the results from the mood-sensitive LLM 132 are sent to the agent 140, which then performs the function of communicating with the user 102.

[0038] Referring to Figure 2, a schematic diagram 200 of the state diagram of a mood and psychological state-aware interaction with a multimodal LLM is shown. Here, LLM mode A (empathetic conversation) 202 serves the purpose of minimizing the distance between the user's mood VAD and the LLM response VAD 212. This is achieved by a module ("empathetic conversation computation module") 216 for calculating the distance between the user's VAD and the LLM response VAD.

[0039] Mode B (Mood Change) 204 begins with explicit user input 208 to determine the current mood state and the desired mood state (specified in fixed or relative VAD regions). A decision 210A is required to determine whether the current mood is equal to the desired mood. If "yes" 210B, Mode C (Mood-Independent Interaction) 206 begins. If "no" 210C, this results in the objective 214 of minimizing the VAD values ​​of the user's current and desired mood states. This is achieved by a module ("Mood Change Calculation Module") 218 ​​for calculating the distance between the desired mood (VAD) and the user's current VAD.

[0040] The timeline 240, showing the process of turn 0 222, includes an input prompt 220, which leads to a recognized mood (VAD) 224, a prompt customization module 226, a calculation of the LLM and its VAD 228, and a retrieval of the user's expressed response to the LLM 230. The output of the recognized mood (VAD) 224 is processed by the empathetic conversation calculation module 216 and the mood change calculation module 218. The output of the calculation of the LLM and its VAD 228 is processed by the empathetic conversation calculation module 216.

[0041] The process of turn 1 238 in timeline 240 includes a second input prompt 232, which leads to a second recognized mood (VAD) 234, which leads to a second prompt customization module 236, and so on. The output of the second recognized mood (VAD) 234 is processed by the mood change calculation module 218. Finally, the output of the user expression response to LLM 230 is processed by the psychological state estimation module 242, which estimates the user's psychological state (e.g., confusion, frustration), and the result is sent to the second prompt customization module 236.

[0042] Referring to Figure 3, a schematic diagram 300 of the computational flow chart for mood and psychological state-aware interaction with a multimodal LLM is shown. In this flow chart, the subscript *g* represents the objective, and the subscript d represents the difference.

[0043] The flowchart begins with a mood-aware target variable module 302 (either relative VAD or fixed-area VAD). The output of this mood-aware target variable module 302 leads to a decision 304A on whether to use relative VAD. If "yes" 304C, a process 308 is performed to calculate the target mood based on the relative target variable, and the user target mood (V g ,A g ,D g ) is sent to module 310. If "No" 304B, the process of setting the user's target mood as a point within the fixed VAD area is performed, and similarly the user target mood (V g ,A g ,D g ) This is sent to module 310. (The subscript 'g' signifies the target value V, A, or D, depending on the case.)

[0044] Separately, there is a measured user mood 314A from turn t=0 (latest) to tn, which is (V t ,A t ,D t )Starting from 314B, then (V t-1 ,A t-1 ,Dt-1 )314C, and (V t-n ,A t-n ,D t-n This includes a series of VAD readings that continue up to 314D. This measurement is from turn t (current turn) to turn tn (the most recent *n* turn). For example, if the current turn index t is 10 and the value of n is 4, the mood is measured for turns 10, 9, 8, 7, and 6. This result is the user target mood (V g ,A g ,D g ) is sent to module 310, and the user target mood (V g ,A g ,D g Along with the output of module 310, it is sent to module 316A, which calculates the mood difference between the user conversation turns and the target mood for all *n* turns. This calculation is performed by (V d ,A d ,D d )=(V t ,A t ,D t )-(V g ,A g ,D g ) Starting from 316B, (V d-n ,A d-n ,D d-n )=(V t-n ,A t-n ,D t-n )-(V g ,A g ,D g )Terminate at 316C. (The subscript *d* means the difference value of V, A, or D.) The result of the calculation is made available so that the LLM can calculate the response 312.

[0045] III. Module Description

[0046] Further explanation of the aforementioned module is provided below.

[0047] A. Speech-to-text module

[0048] The speech-to-text conversion module 110 transcribes the input speech signal into text in near real-time. This module may be implemented using a computationally intensive, edge-based automatic speech recognition model built for real-time inference. Optionally, speech recognition may use visual information to improve its accuracy. Optionally, speech recognition may use visual information only to estimate what was said. Optionally, visual information may be used to recognize the user, process only the user's speech, and ignore speech from others in the recording. Optionally, the user may avoid the need for speech recognition and directly input text using a keyboard. This directly input text, or text transcribed from speech, is first modulated by the user's mood and mental state, and then sent to the LLM as an input prompt.

[0049] B. Prompt

[0050] A prompt consists of conversational messages from the user to the system. A prompt is considered a single conversational turn. A turn may consist of multiple words, sentences, or utterances. In addition to verbal cues, a prompt may include all nonverbal cues (such as facial expressions, tone of voice, and eye movements).

[0051] C. Description of Emotions

[0052] Expressed emotions, or apparent emotions, can sometimes be modeled as points in a three-dimensional continuous space constructed using the dimensions of valence, arousal, and dominance, based on Russell's Circumplex Model of Affect. See James A Russell, A Circumplex Model of Affect, Journal of Personality and Social Psychology Vol.39, No.6, 1161-1178 (1980).

[0053] Russell's model suggests that valence and activity are independent bipolar dimensions. "Independent" means that valence and activity are not correlated. "Bipolar" means that opposing emotional terms represent opposite poles of valence and activity, respectively.

[0054] Other terms used for emotional valence and arousal are pleasure and activation. Optionally, different continuous emotion spaces may be used. Optionally, one-dimensional or two-dimensional subspaces may be used. Optionally, higher-dimensional continuous emotion spaces may be used. Expressed emotion, or apparent emotion, may be estimated using machine learning systems by asking an observer to estimate the emotion, or by asking a person expressing the emotion to rate their own emotion. Expressed emotion may be described as a single value for a specific time unit (e.g., 10 milliseconds) or per conversational unit (e.g., word, sentence, turn, utterance). In this disclosure, “VAD” means any emotion description as defined in this paragraph.

[0055] D. Single Modal Mood Inference Module

[0056] A single modal mood inference module may individually output a temporal aggregation of expressed emotion from each modality—face, voice, text, and physiological signals—to predict the user's apparent and / or perceived mood state. These modality-specific mood predictions may be fused into a single apparent mood state in a multimodal mood fusion module. Alternatively, a person may input a mood into the system. This person may be someone other than the user.

[0057] Mood may be expressed as a value on a VAD scale, or simplified to a one-dimensional scale [-1.0, 1.0]. Mood may be calculated through the temporal aggregation of expressive emotional information embedded in the input prompt. Prompts may include nonverbal data from facial, vocal, and physiological signals.

[0058] To estimate mood from each modality, modality-specific expressive features (e.g., facial action unit intensity, speech low-level descriptors, etc.) may first be extracted, and then VAD scores may be predicted on a given time unit basis (per frame, per word, per sentence, per turn, per utterance). These time unit VAD predictions may then be temporally aggregated into modality-specific mood scores for the entire prompt. The variance of the VAD predictions may be used as a proxy for calculating modality-specific confidence scores for the corresponding mood predictions.

[0059] 1. Expressed emotions from the language module

[0060] Expressive sentiment from language module 120 may be implemented in several ways to estimate VAD in near real-time on consumer hardware such as mobile phones. VAD may be estimated from language using a predefined sentiment-labeled natural language lexicon to predict expressive sentiment values ​​for selected emotionally significant words in text input. VAD may also be estimated from language using a computationally intensive machine learning model that can run in real-time on consumer hardware. This model may take a sequence of word embedding vectors as input to predict expressive sentiment values ​​on either a word-by-word or utterance-by-utterance basis.

[0061] Models may be trained to automatically identify emotional words and fuse them with temporal context in order to improve the accuracy of expressive emotion prediction.

[0062] 2. Emotions expressed from the facial module

[0063] The expressed emotions from the face module 124 are estimated in near real-time on consumer hardware such as mobile phones by analyzing the expressive cues of the user's face in the input video and their temporal dynamics to estimate VAD values. To meet the latency requirements of the proposed system, implementing this module requires a well-optimized facial expression analysis model pipeline consisting of a face bounding box detector, a face landmark detector, a facial expression recognition model for extracting expressive behavioral cues on a fixed time unit basis, and an emotion prediction model for mapping those cues and their temporal dynamics to different points in the VAD. Here, the facial expression recognition model may be implemented either by directly predicting explicit facial muscle activity using action unit intensity, or by predicting facial expression embedding vectors that implicitly encode different expressive cues. Depending on the temporal prediction architecture, the expressed emotion prediction model may take the per-frame output of the facial expression recognition module to compute an expressed emotion score on a fixed time basis (using sequence-to-sequence mapping) or a variable-length basis (e.g., per word, per sentence, or per utterance), measured in milliseconds. Optionally, the module may use visual cues from body posture to improve expressed emotion estimation. Optionally, the module may detect vital signs, including pupil measurement, heart rate, heart rate variability, and respiratory rate, from visual cues to improve expressed emotion estimation.

[0064] 3. Emotions expressed from the voice module

[0065] The emotions expressed from the voice module 122 may be implemented using a simple regression model that takes handcrafted features as input (e.g., low-level speech descriptors (LLDs) such as Mel-frequency cepstrum coefficients (MFCCs)) to estimate the VAD in near real-time on consumer hardware such as mobile phones.

[0066] Speech-based expressive emotion models may be trained to output emotion values ​​on a per-utterance basis. Speech-based expressive emotion models may use deep learning architectures such as Wav2Vec to predict expressive emotion values ​​directly from raw speech waveform input. Speech-based expressive emotion models may use deep neural networks such as transformer models, recurrent neural networks, or other arbitrary artificial neural networks that may be capable of predicting expressive emotion from speech data.

[0067] 4. Emotions expressed from wearable sensor modules

[0068] The expressed emotion from the wearable module 126 aims to infer the user's "perceived" mood state by sensing the physiological arousal level from the heart rate signal and the emotional valence level from the heart rate variability pattern. Given the limitations of face, voice, and text modalities in predicting low-arousal emotional states, expressed emotion prediction from physiological parameters may complement the emotional information extracted from the aforementioned modalities. The expressed emotion prediction performance of the wearable module 126 requires establishing a user-specific "neutral" baseline for these physiological signals for expressed emotion, which may be done individually for each dimension of emotional valence, arousal, and dominance.

[0069] E. Multimodal Mood Fusion Module

[0070] The multimodal mood fusion module 130 may use the confidence scores of these single-modal prediction models to fuse mood predictions from each modality into a single mood score for accurate estimation of the user's mood state. These confidence scores may be used as a surrogate measure indicating the reliability of different modalities for mood sensing. Fusion may be implemented as a weighted average (linear regression) of all single-modal mood estimation modules. Fusion may be implemented as a (recurrent) neural network that takes the outputs of the single-modal mood estimation modules as input. Fusion may be implemented as a support vector machine that takes the outputs of the single-modal mood estimation modules as input. Any of the above methods may use intermediate results of the single-modal mood estimation modules as (additional) input, such as the output of the layer immediately preceding the final layer of the artificial neural network.

[0071] F. Prompt Customization Module

[0072] The prompt customization modules 226 and 236 take the transcribed text output from the speech-to-text module and the multimodal mood score as input. If turn i >= 1, the user's mental state estimated using the mental state estimation module 242 is additionally provided as input to this module.

[0073] Prompt customization modules 226 and 236 encode user mood and psychological state information into text prompts. These modules may be implemented by a rule-based expert system that uses predefined lookup tables to map specific moods and / or psychological states to prompt customization types. The modules also include desired mood goals m t The training set and different mood values ​​(m t -m rThis may be implemented by fine-tuning the weights of a pre-trained LLM using a cost function that penalizes outputs with ). The module has a desired mood goal m t The training set and different mood values ​​(m t -m r This may be implemented by continuing the original LLM with a second LLM that is trained using a cost function that penalizes outputs with ).

[0074] G. Psychological State Estimation Module

[0075] The psychological state estimation module 242 aims to capture the user's emotional response to the LLM output. For example, "agreement," "confusion," and "frustration" are some psychological states commonly encountered in interactions with digital assistants. This may be proposed as a classification model that takes facial and voice input captured during the user's response to the LLM output and returns a list of probability scores indicating the likelihood of different psychological states. Classification may be performed using any machine learning method.

[0076] Optionally, the estimated psychological and mood states may be sent directly to the multimodal LLM, avoiding the need for heuristically designed prompt customization modules 226, 236.

[0077] For LLM to respond quickly in less than 500 milliseconds, it is crucial that mood estimation from high-dimensional input data (video and audio) is implemented in a resource-efficient manner. To speed up the mood inference step, these four single-modal mood modules may be executed in parallel as soon as the required input signals are extracted from a given multimodal input.

[0078] IV. Sample Input and Output

[0079] Table 1 shows exemplary input and output samples illustrating the difference between general-purpose LLM output and mood-sensitive LLM output.

[0080] Table 1: JPEG2026519382000002.jpg223147

[0081] V. Innovation

[0082] The innovations in this disclosure include:

[0083] 1. This system generates a response within 500 milliseconds of the end of a user turn and performs all calculations on consumer hardware, resulting in the low latency required for responsiveness, which leads to a more engaging and empathetic user experience.

[0084] 2. Deriving a user's emotional state using message sentiment metric analysis and / or facial behavior and / or voice parameters leads to a more engaging and empathetic user experience.

[0085] 3. Considering emotional responses to recent LLM prompts leads to a more engaging and empathetic user experience.

[0086] 4. Considering the emotional response to the user's emotional state while providing input to the LLM leads to a more engaging and empathetic user experience.

[0087] 5. Considering the emotional response to the most recent LLM prompt and the emotional state while providing input to the LLM leads to a more engaging and empathetic user experience.

[0088] VI. Additional Disclosures

[0089] The additional disclosures include the following:

[0090] 1. The system includes: The first module set includes: 1) A first input prompt module having a first input prompt module output, 2) A first mood-responsive module that reviews the output of the first input prompt module using emotional valence, arousal level, and dominance, and creates the output of the first mood-responsive module. 3) A first customization prompt module that customizes the output of the first mood-responsive module and creates the output of the first customization prompt module. 4) A first computed large-scale language model (LLM) response module that uses emotional valence, arousal level, dominance, and the output of the first customized prompt module to create a first computed large-scale language model (LLM) response output. 5) A first capture module that determines the user's expressive response to the first computing LLM response output. A psychological state estimator that operates on the first compute LLM response output to produce a psychological state estimation output. The second module set includes: 1) A second input prompt module having a second input prompt module output, 2) A second mood-responsive module that reviews the output of the second input prompt module using emotional valence, arousal level, and dominance, and creates a second mood-responsive module output. 3) A second customization prompt module that customizes a second mood response module output based at least partially on the psychological state estimation output and creates a second customization prompt module output.

[0091] 2. The system described in paragraph 1, wherein the first module set and the second module set are completed within 500 milliseconds.

[0092] 3. The system described in paragraph 2, wherein a first mood-responsive module output and a first computed LLM response output are used to calculate the distance between the user's response valence, arousal level, and dominance and the LLM's response valence, arousal level, and dominance.

[0093] 4. The system described in paragraph 3, wherein the first set of modules and the second set of modules attempt to substantially minimize the distance between the response valence, arousal, and superiority and the response valence, arousal, and superiority of the LLM.

[0094] 5. The system described in paragraph 1, wherein the output of a first mood-responsive module and the output of a second mood-responsive module are used to calculate the distance between the user's current emotional valence, arousal, and dominance and the user's desired emotional valence, arousal, and dominance.

[0095] 6. The system described in paragraph 5, wherein the first mood-responsive module and the second mood-responsive module attempt to minimize the distance between the user's current emotional valence, arousal level, and dominance and the user's desired emotional valence, arousal level, and dominance.

[0096] 7. The system described in paragraph 6, further comprising the current mood state and the desired mood state.

[0097] 8. The system described in paragraph 7, wherein the current mood state is not equal to the desired mood state.

[0098] 9. The methods include: The current turn is one turn prior to at least one previous turn, and the current turn includes: The first emotional valence, arousal level, and dominance value are derived from the user's language, and the user's language is generated from at least one of the user's voice and / or user's text input. The system derives a second emotional valence, arousal level, and dominance value from the user's voice. A third emotional valence, arousal level, and dominance value are derived from the user's face. Deriving a fourth emotional valence, arousal level, and dominance value from the user's wearable device. To generate a multimodal mood fusion value, the user's mood state is combined based on the first emotional valence, arousal level, and dominance value, the second emotional valence, arousal level, and dominance value, the third emotional valence, arousal level, and dominance value, and the fourth emotional valence, arousal level, and dominance value. To generate psychological state estimates, the system estimates the user's psychological state based on the user's voice and face, and Based on the user's estimated mood state and estimated psychological state, a large-scale language model (LLM) is used to generate mood-aware responses for the agent.

[0099] 10. The method described in paragraph 9, wherein the derivation step, the joining step, the estimation step, and the generation step are completed within 500 milliseconds.

[0100] 11. The method described in paragraph 10, wherein the generation step is based on the user's second mood state while providing input prompts to the LLM in the current turn.

[0101] 12. The method described in paragraph 10, wherein the generation step is based on the estimated mood state of the user and the estimated psychological state of the user while the generation step is responding to the LLM's response in at least one previous turn.

[0102] 13. The method described in paragraph 10, wherein the generation step is based on the LLM's response in at least one previous turn.

[0103] 14. The method described in paragraph 10, wherein the generation step is based on a desired mood state and a desired psychological state.

[0104] 15. The method described in paragraph 10, wherein the generation step is based on user mood measurement from at least one previous turn, which is used to establish the current user target mood.

[0105] 16. The method described in paragraph 15, wherein the generation step further depends on calculating the mood difference between at least one previous turn and the current user target mood.

[0106] 17. The method described in paragraph 10, wherein the generation step sets a mood-responsive target using relative emotional valence, arousal level, and dominance value by calculating a user target mood based on relative target variables.

[0107] 18. The method described in paragraph 10, wherein the generation step sets a user target mood using fixed emotional valence, arousal level, and dominance value.

[0108] 19. The method described in paragraph 10, comprising using a predefined emotion-labeled natural language lexicon to predict the expressive emotion value of selected emotionally prominent words, for deriving a first emotional valence, arousal, and dominance value.

[0109] 20. The method described in paragraph 10, further comprising identifying emotional words and fusing them with temporal context for accuracy in predicting expressed emotion.

[0110] 21. A method according to paragraph 10, comprising a facial expression analysis model pipeline using at least one of the following to derive a third emotional valence, arousal, and dominance value from a user's face: a facial boundary box detector, a facial landmark detector, a facial expression recognition model for extracting expressive behavioral cues on a fixed time unit basis, and an emotion prediction model for mapping the behavioral cues to different points on the third emotional valence, arousal, and dominance value.

[0111] 22. The method described in paragraph 10, comprising an expressive emotion prediction module that calculates an expressive emotion score on a fixed-length basis, deriving a third emotional valence, arousal level, and dominance value from the user's face.

[0112] 23. The method described in paragraph 10, comprising an expressive emotion prediction module that calculates an expressive emotion score on a variable length basis, deriving a third emotional valence, arousal level, and dominance value from the user's face.

[0113] 24. A method according to paragraph 10, wherein deriving a third emotional valence, arousal level, and dominance value from the user's face includes detecting at least one of the following from visual cues: pupillary measurement, heart rate, heart rate variability, and respiratory rate.

[0114] 25. The method described in paragraph 10, wherein a second emotional valence, arousal level, and dominance value are derived from the user's voice, and an architecture is used to predict expressed emotional values ​​directly from the voice waveform input.

[0115] 26. The method described in paragraph 10, wherein deriving a fourth emotional valence, arousal level, and dominance value from a user's wearable includes sensing a physiological arousal level from a heart rate signal and an emotional valence level from a heart rate variability pattern.

[0116] 27. The method described in paragraph 26, further comprising establishing a user-specific neutral baseline of physiological signals for expressed emotion.

[0117] 28. The method described in paragraph 10, further comprising a rule-based expert system in which generating mood-responsive responses is implemented via a predefined lookup table for mapping the user's estimated mood state and the user's estimated psychological state to prompt customization types.

[0118] 29. The method described in paragraph 10, further comprising fine-tuning the weights of an LLM using a training set of desired mood values ​​and a cost function that penalizes outputs with different mood values, thereby generating a mood-responsive response.

[0119] 30. The method described in paragraph 10, further comprising fine-tuning the weights of a second LLM using a training set of desired mood values ​​and a cost function that penalizes outputs with different mood values, thereby generating a mood-responsive response.

[0120] VII. Conclusion

[0121] Specific embodiments are described in the aforementioned specification. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention as described in the following claims. Accordingly, the specification and drawings should be viewed in an illustrative rather than restrictive sense, and all such modifications are intended to be within the scope of this teaching.

[0122] Furthermore, in this document, relational terms such as “first” and “second,” “above” and “below” may be used solely to distinguish one entity or action from another, and do not necessarily require or imply any actual relationship or order between such entities or actions. “Comprises,” “comprising,” “has,” “having,” “includes,” “including,” “contains,” “containing,” or any variation thereof are intended to cover non-exclusive inclusion, and a process, method, article, or apparatus that “includes, has, includes, contains” a list of elements may not include only those elements, but may also include elements not explicitly listed, or other elements specific to such process, method, article, or apparatus. Elements described as "comprises...a," "has...a," "includes...a," or "contains...a" do not preclude the presence of additional identical elements in processes, methods, articles, or apparatus that include, have, include, or contain such elements, unless otherwise expressly stated herein. The terms "a" and "an" are defined as one or more unless expressly stated otherwise herein. Terms used as "substantially," "essentially," "approximately," "about," or any variation thereof are defined as being reasonably close to the extent understood by those skilled in the art. The term "combined," as used herein, is defined as being connected, but not necessarily directly or mechanically. A device or structure "configured" in a particular way is configured at least in that way, but may also be configured in ways not listed.

[0123] A summary of the disclosure is provided to allow readers to quickly confirm the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the detailed description of the invention described above, various features are grouped into various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly enumerated in each claim. Rather, as reflected in the following claims, the subject matter of the invention lies in fewer features than all the features of a single disclosed embodiment. Accordingly, the following claims are incorporated into the detailed description of the invention, with each claim existing independently as subject matter claimed individually.

Claims

1. A system, wherein the system is 1) A first input prompt module having a first input prompt module output, 2) A first mood-responsive module that reviews the output of the first input prompt module using emotional valence, arousal level, and dominance, and creates a first mood-responsive module output, 3) A first customization prompt module that customizes the output of the first mood-responsive module and creates a first customization prompt module output, 4) A first computed large language model (LLM) response module that uses emotional valence, arousal level, dominance and the output of the first customized prompt module to create a first computed large language model (LLM) response output, 5) A first capture module that determines the user's expressive response to the first computing LLM response output, A first module set equipped with, A psychological state estimator that operates on the first computing LLM response output to create a psychological state estimation output, 1) A second input prompt module having a second input prompt module output, 2) A second mood-responsive module that reviews the output of the second input prompt module using emotional valence, arousal level, and dominance, and creates a second mood-responsive module output, 3) A second customization prompt module that customizes the output of the second mood-responding module based at least partially on the psychological state estimation output and creates a second customization prompt module output, A second module set equipped with, A system equipped with these features.

2. A system according to claim 1, wherein the first module set and the second module set are configured to complete their operation within 500 milliseconds.

3. The system according to claim 2, wherein the first mood-responsive module output and the first computed LLM response output are used to calculate the distance between the user's response valence, arousal level, and dominance and the LLM's response valence, arousal level, and dominance.

4. A system according to claim 3, wherein the first module set and the second module set substantially minimize the distance between the response valence, arousal level, and superiority and the response valence, arousal level, and superiority of the LLM.

5. A system according to claim 1, wherein the output of the first mood-responsive module and the output of the second mood-responsive module are used to calculate the distance between the user's current emotional valence, arousal level, and dominance and the user's desired emotional valence, arousal level, and dominance.

6. A system according to claim 5, wherein the first mood-responsive module and the second mood-responsive module attempt to minimize the distance between the user's current emotional valence, arousal level, and superiority and the user's desired emotional valence, arousal level, and superiority.

7. The system according to claim 6, further comprising a current mood state and a desired mood state.

8. The system according to claim 7, wherein the current mood state is not equal to the desired mood state.

9. A method that includes the current turn preceding at least one previous turn, The current turn mentioned above is, The process involves deriving a first emotional valence, arousal level, and dominance value from the user's language, wherein the user's language is generated from at least one of the user's voice and the user's text input. A process for deriving a second emotional valence, arousal level, and dominance value from the user's voice, A process for deriving a third emotional valence, arousal level, and superiority value from the user's face, A process for deriving a fourth emotional valence, arousal level, and dominance value from the user's wearable, A step of combining the user's mood state based on the first emotional valence, arousal level, and dominance value, the second emotional valence, arousal level, and dominance value, the third emotional valence, arousal level, and dominance value, and the fourth emotional valence, arousal level, and dominance value in order to generate a multimodal mood fusion value, To generate an estimated psychological state, the process involves estimating the user's psychological state based on the user's voice and the user's face, A step of generating a mood-responsive response for the agent using a large-scale language model (LLM) based on the estimated mood state of the user and the estimated psychological state of the user, A method that includes this.

10. A method according to claim 9, wherein the derivation step, the combination step, the estimation step, and the generation step are completed within 500 milliseconds.

11. A method according to claim 10, wherein the generation step is based on the user's second mood state while the generation step is providing an input prompt to the LLM in the current turn.

12. A method according to claim 10, wherein the generation step is based on the estimated mood state of the user and the estimated psychological state of the user while the generation step is in response to the LLM's response in at least one previous turn.

13. A method according to claim 10, wherein the generation step is based on the response of the LLM in at least one previous turn.

14. A method according to claim 10, wherein the generation step is based on a desirable mood state and a desirable psychological state.

15. A method according to claim 10, wherein the generation step is based on measuring user mood from at least one previous turn, and is used to establish the current user target mood.

16. A method according to claim 15, wherein the generation step further comprises calculating a mood difference between the at least one previous turn and the current user target mood.

17. A method according to claim 10, wherein the generation step calculates a user target mood based on a relative target variable, thereby setting a mood-responding target using relative emotional valence, arousal level, and dominance value.

18. A method according to claim 10, wherein the generation step sets a user target mood using fixed emotional valence, arousal level, and dominance value.

19. A method according to claim 10, comprising the step of using a predefined emotion-labeled natural language lexicon to predict the expressive emotion value of selected emotionally prominent words, for deriving the first emotional valence, arousal, and dominance value.

20. A method according to claim 10, further comprising identifying emotional words and fusing them with a temporal context for accuracy in predicting expressed emotion.

21. A method according to claim 10, wherein deriving the third emotional valence, arousal, and dominance values ​​from the user's face comprises a facial expression analysis model pipeline using at least one of: a facial boundary box detector, a facial landmark detector, a facial expression recognition model for extracting expressive behavioral cues on a fixed time unit basis, and an emotion prediction model for mapping the behavioral cues to different points on the third emotional valence, arousal, and dominance values.

22. A method according to claim 10, comprising an expressive emotion prediction module that derives the third emotional valence, arousal level, and dominance value from the user's face, and calculates an expressive emotion score on a fixed-length basis.

23. A method according to claim 10, comprising an expressive emotion prediction module that calculates an expressive emotion score on a variable length basis, deriving the third emotional valence, arousal level, and dominance value from the user's face.

24. A method according to claim 10, wherein deriving the third emotional valence, arousal level, and dominance value from the user's face includes the step of detecting at least one of the following from visual cues: pupillary measurement, heart rate, heart rate variability, and respiratory rate.

25. A method according to claim 10, wherein the second emotional valence, arousal level, and dominance value are derived from the user's voice, and the architecture is used to predict expressed emotional values ​​directly from a voice waveform input.

26. A method according to claim 10, wherein deriving the fourth emotional valence, arousal level, and dominance value from the user's wearable includes the steps of sensing a physiological arousal level from a heart rate signal and an emotional valence level from a heart rate variability pattern.

27. A method according to claim 26, further comprising the step of establishing a user-specific neutral baseline of physiological signals for expressed emotion.

28. A method according to claim 10, further comprising a rule-based expert system for generating the mood-responsive response, which is implemented via a predefined lookup table for mapping the user's estimated mood state and the user's estimated psychological state to prompt customization types.

29. A method according to claim 10, further comprising the step of fine-tuning the weights of the LLM using a training set of desired mood values ​​and a cost function that penalizes outputs having different mood values, for generating the mood-responsive response.

30. A method according to claim 10, further comprising the step of fine-tuning the weights of a second LLM using a training set of desired mood values ​​and a cost function that penalizes outputs having different mood values, for generating the mood-responsive response.

Images