Intelligent agent humanization interaction system and method based on multi-modal cognitive graph and large model middleware

Abstract

The application provides an agent personification interaction system and method based on a multi-modal cognitive graph and a large model middleware, comprising: collecting current interaction data of a user; inputting the current interaction data into a pre-constructed multi-modal cognitive graph for context analysis and generating a corresponding structured context description vector; inputting the structured context description vector into a large model middleware to generate an initial response content matched with an explicit intention and an implicit emotional state; decomposing the initial response content into a speech synthesis parameter, a facial expression driving parameter and a motion posture driving parameter; generating a personified voice according to the speech synthesis parameter, while driving a preset three-dimensional virtual image to make coordinated expressions and actions according to the facial expression driving parameter and the motion posture driving parameter, forming a multi-modal synchronous output, and outputting a personification quality score of this interaction. The application can realize natural, smooth and highly personified interaction.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and human-computer interaction technology, and more specifically, to an intelligent agent anthropomorphic interaction system and method based on multimodal cognitive graphs and large model middleware. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent dialogue systems, virtual assistants, and digital humans have been widely used in customer service, education, entertainment, and other fields. In existing technologies, the interaction between intelligent agents and users is mostly limited to single-modal responses of text or voice, lacking the ability to output multimodal functions such as facial expressions, body movements, and vocal rhythms. This results in a stiff interaction process, a lack of emotional warmth, and a low degree of "humanization."

[0003] In recent years, some technologies have attempted to drive virtual avatars using predefined facial expression templates or action libraries. However, facial expressions and voice content lack dynamic correlation, actions and semantic stress are difficult to synchronize, and response strategies cannot be adjusted according to the user's real-time emotional state. Furthermore, most systems lack the ability to perceive intention deviations during interaction and the capacity for online learning, making it difficult to create personalized and continuously optimized interactive experiences.

[0004] Multimodal cognitive graphs can integrate heterogeneous modal information, but existing graph reasoning is mainly based on static knowledge and lacks deep integration with large language models. Although large language models have powerful generation capabilities, directly generating text as multimodal driving signals still has problems such as difficulty in parameter decoupling and cross-modal alignment differences. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of the prior art and provide an intelligent agent anthropomorphic interaction system and method based on multimodal cognitive graph and large model middleware, so as to solve the technical problems of low degree of anthropomorphism in intelligent agent interaction, lack of coordination of multimodal output, inability to dynamically perceive user emotions and intention deviations, and lack of personalized online learning capabilities in the prior art.

[0006] In a first aspect, the present invention provides an anthropomorphic interaction method for intelligent agents based on multimodal cognitive graphs and large model middleware, the method comprising: In response to user-triggered interaction requests, collect the user's current interaction data; The current interaction data is input into a pre-constructed multimodal cognitive graph for context analysis, which identifies the user's explicit intentions and implicit emotional states, and generates corresponding structured context description vectors. The structured context description vector is input into the large model middleware, which calls a pre-set anthropomorphic strategy template to generate initial response content that matches the explicit intent and the implicit emotional state. The initial response content is decomposed into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters through the parameterized adaptation layer built into the large model middleware. Based on the speech synthesis parameters, an anthropomorphic voice is generated, and at the same time, based on the facial expression driving parameters and the action posture driving parameters, a preset three-dimensional virtual image is driven to make coordinated expressions and actions, forming a multimodal synchronous output; During and after the multimodal synchronous output, user feedback interaction data is continuously collected. The confidence level of the feedback interaction data is evaluated through the multimodal cognitive map to generate an interaction deviation index. By integrating the structured context description vector, the initial response content, and the interaction deviation index, the weights of the intent-response path in the multimodal cognitive graph are dynamically updated using a reinforcement learning algorithm, and a human-like quality score for this interaction is output.

[0007] Preferably, the step of inputting the current interaction data into a pre-constructed multimodal cognitive graph for contextual analysis to identify the user's explicit intentions and implicit emotional states includes: The current interactive data is decomposed into three types of modal nodes: speech features, image features, and text features; Cross-modal link retrieval is performed on the multimodal cognitive graph using the three types of modal nodes; If a matching path is found that is directly related to the three types of modal nodes, the target intent node at the end of the matching path is taken as the explicit intent, and the sentiment tag associated with the target intent node is taken as the implicit sentiment state. If no direct matching path is found, the graph inference engine is activated, and the association probability with each candidate intent node is calculated through the graph attention network. Candidate intents with association probabilities exceeding a preset threshold are selected as the explicit intents, and the corresponding sentiment tags are output synchronously.

[0008] Preferably, the large model middleware calls a pre-set anthropomorphic strategy template to generate initial response content that matches the explicit intention and the implicit emotional state, including: The emotional tone of the response is determined based on the implicit emotional state, and the emotional tone includes pleasure, empathy, concern, or neutrality; The functional content of the response is determined based on the explicit intent, and the functional content includes information query, task execution, emotional companionship or casual conversation; The emotional tone and the functional content are concatenated into a synthetic prompt word, which is then input into the large language model in the large model middleware to generate the initial response content.

[0009] Preferably, when inputting into the large language model in the large model middleware, the temperature parameter is set to 0.7; after generating the initial response content, the initial response content is further filtered for sensitive words and its length is pruned to ensure that it meets the preset security specifications and output duration constraints.

[0010] Preferably, the step of decomposing the initial response content into speech synthesis parameters, facial expression driving parameters, and motion posture driving parameters through the parameterized adaptation layer built into the large model middleware includes: The text of the initial response is input into the text encoder of the parameterized adaptation layer to obtain the text embedding vector; The text embedding vector is mapped to the speech synthesis parameter space, the facial blending shape parameter space, and the skeletal animation parameter space respectively to output the speech synthesis parameters, the facial expression driving parameters, and the motion posture driving parameters. The speech synthesis parameters include phoneme duration, fundamental frequency curve, and spectral envelope; the facial expression driving parameters include mouth opening and closing degree, eyebrow raising angle, and eyelid closing degree; the action posture driving parameters include head turning angle, gesture trajectory, and torso swaying amplitude.

[0011] Preferably, the step of generating anthropomorphic speech based on the speech synthesis parameters, and simultaneously driving the 3D virtual avatar to perform coordinated expressions and movements based on the facial expression driving parameters and the motion posture driving parameters, specifically includes: Based on the phoneme duration and the spectral envelope, an anthropomorphic speech is generated by a speech synthesis engine, and the anthropomorphic speech is segmented into multiple speech segments according to the phoneme duration boundaries; During the playback of each of the aforementioned voice segments, the lip mesh vertex of the three-dimensional virtual image is displaced according to the degree of mouth opening and closing, while the eyebrow and eye movements are driven according to the eyebrow raising angle and the eyelid closing degree. The upper limbs and torso of the three-dimensional virtual image are driven according to the gesture trajectory and the torso swing amplitude, and the peak point of the gesture trajectory is aligned with the stress point of the anthropomorphic voice; the cervical skeleton of the three-dimensional virtual image is driven according to the head tilt angle. At the interval between two adjacent speech segments, or during the silence segment of the anthropomorphic speech, the gaze and blinking behavior of the three-dimensional virtual avatar are inserted, wherein the gaze is directed at the user's position, and the blinking behavior is spaced 6 to 8 seconds apart. During the playback of the anthropomorphic voice, the instantaneous slope of the fundamental frequency curve is extracted in real time; when the instantaneous slope exceeds a preset steep change threshold, the eyebrows of the three-dimensional virtual image are driven to quickly rise once and remain for 0.3 seconds before resetting to simulate a surprised or emphasized expression.

[0012] Preferably, it further includes: Record the structured context description vector, the initial response content, the interaction deviation index, and the anthropomorphic quality score generated for each interaction to form an interaction history dataset; When the number of dialogue rounds for the same user in the interaction history dataset exceeds a preset round threshold, a pre-set personalized fine-tuning model is invoked. Based on the interaction history dataset, the node weights in the multimodal cognitive graph and the low-rank adaptation parameters of the large model middleware are incrementally updated to generate a user-specific anthropomorphic interaction configuration.

[0013] Secondly, this invention provides an intelligent agent anthropomorphic interaction system based on a multimodal cognitive graph and a large model middleware, comprising: The data acquisition module is used to collect the user's current interaction data in response to the user's interactive request. The structured context description vector generation module is used to input the current interaction data into a pre-constructed multimodal cognitive graph for context analysis, identify the user's explicit intentions and implicit emotional states, and generate corresponding structured context description vectors. The initial response content generation module is used to input the structured context description vector into the large model middleware. The large model middleware calls a preset anthropomorphic strategy template to generate initial response content that matches the explicit intention and the implicit emotional state. The initial response content decomposition module is used to decompose the initial response content into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters through the parameterized adaptation layer built into the large model middleware. The multimodal synchronous output module is used to generate human-like speech based on the speech synthesis parameters, and simultaneously drive a preset three-dimensional virtual image to make coordinated expressions and movements based on the facial expression driving parameters and the action posture driving parameters, thereby forming multimodal synchronous output. The interaction deviation index generation module is used to continuously collect user feedback interaction data during and after the multimodal synchronous output, evaluate the confidence level of the feedback interaction data through the multimodal cognitive map, and generate an interaction deviation index. The anthropomorphic quality score output module is used to integrate the structured context description vector, the initial response content, and the interaction deviation index, dynamically update the weights of the intent-response path in the multimodal cognitive graph through a reinforcement learning algorithm, and output the anthropomorphic quality score of this interaction.

[0014] Thirdly, the present invention provides a readable medium including executable instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform any of the methods described in the first aspect.

[0015] Fourthly, the present invention provides an electronic device including a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor performs the method as described in any of the first aspects.

[0016] This invention provides an intelligent agent anthropomorphic interaction system and method based on a multimodal cognitive graph and a large model middleware. By constructing a deeply coupled architecture of multimodal cognitive graph and large model middleware, it first achieves multimodal contextual understanding of user speech, images, and text, accurately extracting explicit intentions and implicit emotional states, laying the cognitive foundation for anthropomorphic responses. Second, it utilizes the large model middleware combined with anthropomorphic strategy templates to generate emotionally rich initial responses, and decouples these responses into speech parameters, facial parameters, and action parameters through a parameterized adaptation layer, achieving end-to-end mapping from text to multimodal driving signals. Third, in the interaction output stage, speech generation is driven by speech parameters such as phoneme duration and spectral envelope, simultaneously based on lip shape and eyebrow... Parameters such as hair, eyelids, gestures, torso, and head rotation drive the 3D virtual avatar. It innovatively utilizes pauses in speech to insert natural gaze and blinking behaviors, and uses the slope of the fundamental frequency curve to trigger rapid eyebrow raising to simulate surprise / emphasis expressions. This achieves a high degree of synergy between speech and facial expressions in terms of timing, semantics, and prosody, significantly enhancing the anthropomorphic realism of the interaction. Furthermore, by continuously collecting user feedback and generating interaction deviation indicators, combined with reinforcement learning to dynamically update the intent-response path weights in the cognitive graph, the system can perceive online whether its responses meet user expectations and continuously optimize. Finally, a personalized fine-tuning mechanism based on historical datasets can generate exclusive interaction configurations for different users, further improving the adaptability and user stickiness of long-term interactions.

[0017] The further effects of the aforementioned non-conventional preferred method will be explained below in conjunction with specific embodiments. Attached Figure Description

[0018] To more clearly illustrate the embodiments of the present invention or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of an anthropomorphic interaction method for intelligent agents based on multimodal cognitive graphs and large model middleware, provided in an embodiment of the present invention. Figure 2 A schematic diagram of another intelligent agent anthropomorphic interaction method based on multimodal cognitive graph and large model middleware provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the composition of an intelligent agent anthropomorphic interaction system based on a multimodal cognitive graph and a large model middleware, according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] See Figure 1 The image shows a specific embodiment of an anthropomorphic interaction method for intelligent agents based on a multimodal cognitive graph and a large model middleware, provided by the present invention. In this embodiment, the anthropomorphic interaction method for intelligent agents based on a multimodal cognitive graph and a large model middleware includes: Step 101: In response to the user's triggered interaction request, collect the user's current interaction data; Specifically, in this embodiment, the interaction request is an event signal generated when a user initiates the intelligent agent interaction service through voice wake-up, touch click, or visual detection. Current interaction data refers to the multimodal raw information generated by the user under this interaction request, including but not limited to the user's voice stream, user facial images or video sequences, and text commands input or recognized by the user. In this step, the system acquires the above data by calling the terminal's microphone, camera, or text input interface and temporarily stores it in a circular buffer for subsequent processing. As the input source for subsequent context analysis, the completeness and real-time nature of the current interaction data directly affect the accuracy of intent recognition.

[0022] Step 102: Input the current interaction data into the pre-constructed multimodal cognitive graph for context analysis, identify the user's explicit intentions and implicit emotional states, and generate corresponding structured context description vectors; This step is further divided into two sub-steps: multimodal feature extraction and multimodal cognitive graph reasoning. The two sub-steps are logically connected, but each uses an independent model and data structure.

[0023] Multimodal feature extraction. The system feeds the acquired raw speech stream, facial image sequence, and text commands into three independent pre-trained encoders, converting them into fixed-dimensional feature vector representations. Specifically: Speech feature extraction employs a pre-trained wav2vec 2.0 model (12-layer Transformer architecture, 768-dimensional hidden layers, outputting frame-level acoustic embedding vectors). The output frame-level embeddings are then average-pooled along the time axis to obtain a 768-dimensional speech sentence-level feature vector.

[0024] Image feature extraction was performed using the OpenFace 2.0 toolkit, extracting 17-dimensional facial action units (representing the intensity of local muscle movements such as frowning and smiling) and 3-dimensional head pose Euler angles (pitch, yaw, roll) from user facial image sequences. These 20-dimensional values ​​were then concatenated to form the image feature vector.

[0025] Text feature extraction employs a BERT-base-chinese pre-trained model (12-layer Transformer, 768-dimensional hidden states). User-input text commands or text transcribed from speech recognition are processed by the BERT word segmenter to obtain a word sequence, and the output embedding of the [CLS] token is used as a 768-dimensional text feature vector.

[0026] The three feature vectors (768-dimensional speech, 20-dimensional image, and 768-dimensional text) are concatenated to form a 1556-dimensional multimodal feature matrix, which serves as the initial attributes of the modal nodes in the subsequent graph inference process. All three models are open-source pre-trained models that remain frozen during inference without undergoing fine-tuning.

[0027] Multimodal cognitive graph reasoning. The system calls a pre-built multimodal cognitive graph and a graph attention network deployed on it for context parsing. This multimodal cognitive graph is a heterogeneous directed knowledge graph, whose node types include: modality nodes (the storage location of the speech, image, and text feature vectors generated in the previous sub-step), intent nodes (predefined intent categories, including information query, task execution, emotional companionship, casual conversation, complaint, praise, farewell, etc.), emotion nodes (emotion categories, including happiness, sadness, anger, surprise, fear, disgust, neutrality, and composite emotions represented by a two-dimensional continuous space of valence-arousal), and response template nodes (pre-set response strategy anchors). Edge types include: modality-intent association edges (representing the co-occurrence relationship between a certain modality feature combination and a specific intent), intent-emotion association edges (representing the emotional tendency usually associated with a certain intent), and emotion-response strategy edges (representing the response strategy applicable under a certain emotional tendency). A graph attention network (GAT) is deployed on the multimodal cognitive graph for reasoning. The architecture of this GAT consists of two layers of graph convolutions: the first layer has an input node feature dimension of 1556 (i.e., the feature dimension of the modality node; intent nodes, sentiment nodes, and template nodes are initially randomly orthogonally initialized), and a hidden layer dimension of 256, using four attention heads; the second layer outputs the hidden representation to the predicted probabilities (20-dimensional) of each intent node. ELU activation and Dropout (rate=0.2) are used between layers. During training, a manually annotated multimodal dialogue dataset containing 200,000 samples is used. Each sample records the original signals of three modalities (speech, image, and text) and their corresponding intent and sentiment labels. Data annotation is performed by trained annotators, with a consistency check pass rate higher than 0.85. Training uses the Adam optimizer with a learning rate of 0.001, a batch size of 64, 50 training epochs, and cross-entropy loss.

[0028] During inference, the generated 1556-dimensional multimodal feature matrix is ​​used as the initial attribute input to the GAT for modal nodes. The GAT performs message propagation on the graph structure and outputs the association probability of each candidate intent node. If the probability of an intent node exceeds a preset threshold (0.75 in this embodiment), the node with the highest probability is selected as the explicit intent; if the probabilities of all intent nodes are below 0.75, the backoff mechanism in the graph is activated: the Softmax distribution of the probabilities of all intent nodes is calculated using a graph attention network, and the top-1 is taken as the explicit intent, while the confidence level corresponding to the intent is output. The implicit sentiment state is obtained by: along the intent-sentiment association edge of the selected explicit intent node, obtaining the sentiment node with the strongest connection (the edge weight is the largest), and using the category label of the sentiment node as the implicit sentiment state; if there is no explicit connection, "neutral" is output by default.

[0029] Finally, a structured context description vector is generated: explicit intentions are encoded as 20-dimensional one-hot vectors, and implicit emotional states are mapped to 2-dimensional valence-arousal continuous values ​​(e.g., "happy" corresponds to [0.85, 0.7], and "angry" corresponds to [-0.6, 0.8]). These values, along with the confidence values ​​(0~1) output by GAT, are concatenated to form a 23-dimensional dense vector. This vector serves as the standardized input for subsequent large model middleware.

[0030] Step 103: Input the structured context description vector into the large model middleware. The large model middleware calls the preset anthropomorphic strategy template to generate initial response content that matches the explicit intention and implicit emotional state. Furthermore, the large model middleware is a functional component that encapsulates a large language model and its supporting management services. In this embodiment, LLaMA-2 13B or ChatGLM3 is used as the base model, with 13 billion parameters, and a Transformer decoder architecture is adopted. The anthropomorphic strategy template is a set of predefined prompt word construction rules, whose function is to map the discrete intentions and continuous emotions in the structured context description vector into natural language instructions. Specifically: First, the emotional tone of the response is determined based on the valence-arousal coordinates of the implicit emotional state: valence > 0.5 and arousal > 0.5 correspond to "pleasure," valence < 0.3 and arousal > 0.5 correspond to "concern," valence < 0.3 and arousal < 0.3 correspond to "empathy," and the rest correspond to "neutrality." Secondly, the functional content of the response is determined based on the explicit intent: if the intent label is "information query," the functional content is to retrieve and provide information; if it is "task execution," the functional content is to confirm and execute the operation; if it is "emotional companionship," the functional content is to listen and provide emotional feedback; if it is "casual chat," the functional content is to engage in open-domain chat; and if it is "complaint," the functional content is to soothe and resolve the problem. Then, the emotional tone (e.g., "concern") and the functional content (e.g., "soothing and problem-solving") are concatenated into a synthetic prompt word. For example, the format is: "You are an intelligent assistant with the emotion {emotional tone}, and the user needs {functional content}. Please generate an answer that matches this emotion and function:". This prompt word is input into a large language model, with a temperature parameter set to 0.7 (to control generation randomness), a top_p sampling rate of 0.9, and a generation length limit of 150 tokens to obtain the initial response text. After generation, the text is filtered for sensitive words (using a dictionary-based DFA filtering algorithm) and its length is pruned (ensuring the output duration does not exceed 8 seconds), outputting a safe, compliant, and appropriately long initial response content.

[0031] Step 104: Using the parameterized adaptation layer built into the large model middleware, the initial response content is decomposed into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters. Furthermore, the parameterized adaptation layer is a multi-task deep learning module that takes text as input and outputs various continuous control parameters. The core architecture of this adaptation layer includes a text encoder (6-layer Transformer, 768-dimensional hidden states), followed by three parallel mapping heads—a speech parameter head, a face parameter head, and a motion parameter head. Each mapping head is a 2-layer fully connected network (output dimensions T, F, and G, respectively). Training data comes from a large-scale multimodal dialogue corpus, where each sentence of text is annotated with corresponding phoneme durations (generated by Montreal Forced Aligner), fundamental frequency curves (extracted by WORLD vocoder), spectral envelopes (analyzed by STRAIGHT), mouth opening / closing angles (labeled by FaceWarehouse), eyebrow raising angles and eyelid closing angles (detected by OpenFace 2.0), head yaw angles, gesture trajectories, and torso sway amplitudes (motion capture data provided by the CMUpanoptic dataset). The training loss is mean squared error plus a regularization term.

[0032] In actual reasoning, the initial response text is segmented into words and input into a text encoder to obtain a text embedding vector. This vector is then input into three mapping heads, which output values ​​in the corresponding parameter spaces: the speech synthesis parameter space outputs phoneme duration (T-dimensional sequence), fundamental frequency curve (continuous value curve), and spectral envelope (Mel spectrum parameters); the facial blending shape parameter space outputs mouth opening degree (0-1 continuous value), eyebrow raising angle (0-1), and eyelid closure degree (0-1); and the skeletal animation parameter space outputs head deflection angle (Euler angle), gesture trajectory (3D spatial path point), and torso swing amplitude (waist rotation arc). These parameters together constitute the multi-dimensional control signals driving the virtual avatar.

[0033] Step 105: Generate human-like speech based on speech synthesis parameters, and simultaneously drive the preset 3D virtual image to make coordinated expressions and movements based on facial expression driving parameters and action posture driving parameters, forming multimodal synchronous output; Furthermore, this step achieves precise timing alignment between speech and animation. First, based on phoneme duration and spectral envelope, a human-like speech waveform is generated using a speech synthesis engine (such as Tacotron2 + WaveGlow or VITS). The human-like speech is then segmented into multiple speech segments according to phoneme duration boundaries, with each segment corresponding to a phoneme unit. The speech synthesis engine has a sampling rate of 22.05kHz and outputs a PCM audio stream.

[0034] During the playback of each voice segment, the lip mesh vertices of the 3D virtual avatar are displaced based on the degree of mouth opening and closing—specifically through linear blending shape weighting, so that the degree of opening and closing of the upper and lower lips corresponds linearly to the opening and closing value. At the same time, the eyebrow and eye movements are driven based on the eyebrow raising angle and eyelid closing degree: the eyebrow raising angle is mapped to the vertical offset of the eyebrow controller, and the eyelid closing degree is mapped to the rotation angle of the upper and lower eyelids.

[0035] The upper limbs and torso of the 3D virtual avatar are driven by gesture trajectories and torso swing amplitudes: the gesture trajectory is generated by interpolating Bezier curves to create the path of the arm joints, and the peak point of the gesture trajectory (farthest from the starting point) is aligned with the stress point of the anthropomorphic speech (detected by the speech amplitude envelope or fundamental frequency abrupt change point), so that the gesture reaches its maximum amplitude when emphasizing words; the torso swing amplitude drives the smooth rotation of the spinal joints. The cervical skeleton of the 3D virtual avatar is driven by the head tilt angle, allowing the head to rotate naturally.

[0036] To simulate the breathing and gaze patterns in natural human interaction, gaze and blinking behaviors of a 3D virtual avatar are inserted at the intervals between two adjacent speech segments (i.e., micro-pauses at phoneme boundaries) or during silent segments of anthropomorphic speech (where the speech energy is determined by voice energy detection to be below a threshold of -40dB and lasts for more than 30ms). The gaze behavior is constrained by the eyeball skeleton to point to the user's head position in 3D space; the blinking behavior is a closed-open cycle with intervals of 6 to 8 seconds (randomly and uniformly distributed).

[0037] Furthermore, during the playback of anthropomorphic speech, the instantaneous slope of the fundamental frequency curve is extracted in real time (the difference between the fundamental frequency value of the current frame and the previous frame divided by the time difference). When the instantaneous slope exceeds a preset steep change threshold (e.g., 20 Hz / second), it indicates a sudden pragmatic emphasis or surprise in the speech. At this point, the eyebrows of the 3D virtual avatar are driven to quickly rise once; that is, the eyebrow controller jumps from the current value to the maximum value within 80ms, holds for 0.3 seconds, and then resets. This action simulates the micro-facial expressions of humans when expressing surprise or key emphasis, significantly enhancing the realism of the interaction.

[0038] Step 106: During and after multimodal synchronous output, continuously collect user feedback interaction data, evaluate the confidence level of the feedback interaction data through multimodal cognitive mapping, and generate an interaction deviation index. Furthermore, the feedback interaction data includes subsequent speech, facial expression changes, gaze direction changes, or text input during the user's dialogue turn. The system extracts these features and performs graph inference in the same manner as in step 102 to obtain the user's current actual intent and emotional state. Confidence assessment refers to matching the features of the user's feedback with the expected intent / emotional nodes that the user should exhibit after the ideal response in the graph (the expected nodes are automatically set by the system based on the response content when generating the response), and calculating a matching degree score (0~1). The interaction deviation index is defined as 1 minus this confidence score, with a value range of 0~1. The higher the value, the more inconsistent the user's actual feedback is with the system's expectations, meaning that the current response content may not meet the user's needs.

[0039] Step 107: Integrate the structured context description vector, initial response content, and interaction deviation index, dynamically update the weights of the intent-response path in the multimodal cognitive graph using a reinforcement learning algorithm, and output the anthropomorphic quality score of this interaction.

[0040] Furthermore, this step employs the Q-learning reinforcement learning framework. The state is defined as the concatenation of the current structured context description vector and the response content from the previous round, and the action is the adjustment amount (a continuous value) to the weight of a certain intent-response path in the graph. The reward function is a negative interaction deviation index (i.e., the lower the deviation, the higher the reward). After each interaction, the system adjusts the weights of relevant edges in the graph based on the actual reward obtained, making it more inclined to choose response paths that historically resulted in low deviation in similar future contexts. Finally, the anthropomorphic quality score of this interaction is output. This score is equal to 1 minus the weighted sum of the interaction deviation index and an empirical baseline coefficient (0.2), serving as a quantitative evaluation of the overall naturalness of this interaction.

[0041] As can be seen from the above technical solutions, the beneficial effects of this embodiment are: it constructs a complete closed loop from multimodal perception, feature extraction, graph reasoning, response generation, parameter decoupling to multimodal collaborative output and online closed-loop optimization, which significantly improves the anthropomorphism and real-time adaptive capability of the intelligent agent in the interaction process with the user.

[0042] Figure 1 The embodiments shown are merely basic examples of the method of the present invention. Other preferred embodiments of the method can be obtained by making certain optimizations and extensions based on them.

[0043] like Figure 2 The image shown is another specific embodiment of the intelligent agent anthropomorphic interaction method based on multimodal cognitive graphs and large model middleware according to the present invention. This embodiment further describes the method based on the foregoing embodiments, and includes the following steps: Step 201: Record the structured context description vector, initial response content, interaction deviation index, and anthropomorphic quality score generated for each interaction to form an interaction history dataset; Specifically, after each complete user-agent interaction round, the system stores the following data items generated in the preceding steps in a structured manner: a structured context description vector (recording the user's intent and emotions at that time), initial response content (text format), interaction deviation index (numerical), and anthropomorphic quality score (numerical), along with a timestamp and user identification (anonymized device ID or session ID). All data items are written to a local database or cloud storage in JSON or Parquet format, forming an interaction history dataset. This dataset accumulates continuously as the user's usage frequency increases, recording the user's interaction preferences in different scenarios and emotional states.

[0044] Step 202: When the number of dialogue rounds for the same user in the interaction history dataset exceeds the preset round threshold, the pre-set personalized fine-tuning model is invoked. Based on the interaction history dataset, the node weights in the multimodal cognitive graph and the low-rank adaptation parameters of the large model middleware are incrementally updated to generate a user-specific anthropomorphic interaction configuration.

[0045] Furthermore, in this embodiment, the preset round threshold is set to 30 rounds. This value is based on empirical statistics, as user behavior patterns tend to stabilize after more than 30 rounds of interaction, providing the signal-to-noise ratio conditions for personalized fine-tuning. The personalized fine-tuning model includes two components: a weight fine-tuner for the multimodal cognitive graph, and a low-rank adaptation (LoRA) module for the large model middleware.

[0046] Specifically, for the multimodal cognitive graph, the system extracts frequently occurring intent-emotion combinations and their corresponding low-deviation response paths from the interaction history dataset. An incremental graph learning algorithm (such as Inductive Graph Neural Network, GraphSAGE) is used to locally update the edge weights in the graph: whenever a user expresses a certain intent and emotion and receives a low-deviation feedback score below 0.2, the weight of that intent-response path is increased by 0.05 (with an upper limit of 1.0). For the large model middleware, the system uses user text input from the interaction history dataset and system-generated response text rated as high quality (humanization quality score > 0.85) as fine-tuning samples to fine-tune the original large language model using LoRA. The principle of LoRA is to add a low-rank decomposition matrix A·B next to the attention weight matrix of the Transformer layer, where the parameters of A and B are much fewer than the original model. Training is performed on a small amount of data for 2-3 epochs with a learning rate of 1e-4. The trained LoRA adapter weights are merged and stored with the original model weights to form a user-specific model copy. The user-specific anthropomorphic interaction configuration includes the updated graph weight file and LoRA adapter file. For the user's subsequent interactions, the system will prioritize loading this configuration, enabling the agent to generate responses and multimodal performance styles that better suit the user's individual habits in similar situations (for example, if the user prefers humorous responses, the model will tend to generate humorous statements, along with more exaggerated gestures), thereby achieving a personalized anthropomorphic experience in long-term interactions.

[0047] As can be seen from the above technical solutions, the beneficial effects of this embodiment are: by implementing low-cost incremental personalized fine-tuning based on long-term dialogue accumulation, the intelligent agent can continuously adapt to individual user differences, and achieve continuous evolution and personalized customization of the human-like interaction effect without retraining the full model.

[0048] This invention also provides an anthropomorphic interactive system for intelligent agents based on a multimodal cognitive graph and a large model middleware. See also Figure 3 The image shown is a specific embodiment of an anthropomorphic interactive system for intelligent agents based on a multimodal cognitive graph and a large model middleware, provided by the present invention. This embodiment of the system is used to execute... Figures 1-2 The physical apparatus of the method. Its technical solution is essentially the same as the embodiments described above, and the corresponding descriptions in the embodiments above also apply to this embodiment. The system includes: The data acquisition module 301 is configured to collect the user's current interaction data in response to an interaction request triggered by the user. The structured context description vector generation module 302 is configured to input the current interaction data into a pre-built multimodal cognitive graph for context analysis, identify the user's explicit intentions and implicit emotional states, and generate corresponding structured context description vectors. The initial response content generation module 303 is configured to input the structured context description vector into the large model middleware, and the large model middleware calls the preset anthropomorphic strategy template to generate initial response content that matches the explicit intention and implicit emotional state. The initial response content decomposition module 304 is configured to decompose the initial response content into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters through the parameterized adaptation layer built into the large model middleware. The multimodal synchronous output module 305 is configured to generate human-like speech based on speech synthesis parameters, and simultaneously drive a preset three-dimensional virtual image to make coordinated expressions and movements based on facial expression driving parameters and motion posture driving parameters, thus forming multimodal synchronous output. The interaction deviation index generation module 306 is configured to continuously collect user feedback interaction data during and after multimodal synchronous output, evaluate the confidence of the feedback interaction data through multimodal cognitive graph, and generate an interaction deviation index. The anthropomorphic quality score output module 307 is configured to integrate the structured context description vector, the initial response content, and the interaction deviation index, dynamically update the weights of the intent-response path in the multimodal cognitive graph through a reinforcement learning algorithm, and output the anthropomorphic quality score of this interaction.

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0050] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, and other types. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0051] Memory is used to store instructions for execution. Specifically, instructions for execution are computer programs that can be executed. Memory can include main memory and non-volatile memory, and it provides the processor with execution instructions and data.

[0052] In one possible implementation, the processor reads the corresponding execution instructions from non-volatile memory into main memory and then executes them. Alternatively, it can obtain the corresponding execution instructions from other devices to logically form an anthropomorphic intelligent agent interaction device based on a multimodal cognitive graph and a large model middleware. The processor executes the execution instructions stored in the memory to implement the anthropomorphic intelligent agent interaction method based on a multimodal cognitive graph and a large model middleware provided in any embodiment of the present invention.

[0053] The above is as described in the present invention. Figure 3The method for executing an intelligent agent anthropomorphic interaction system based on multimodal cognitive maps and large model middleware provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed through integrated logic circuits in the processor's hardware or through software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0054] The steps of the method disclosed in the embodiments of this invention can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0055] This invention also proposes a readable medium storing execution instructions. When these instructions are executed by the processor of an electronic device, the device can perform an anthropomorphic interaction method for intelligent agents based on a multimodal cognitive graph and a large model middleware, as provided in any embodiment of this invention. Specifically, it is used to perform actions such as... Figure 1 , Figure 2 The method shown.

[0056] The electronic devices in the foregoing embodiments may be computers.

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can be implemented in a completely hardware embodiment, a completely software embodiment, or a combination of software and hardware.

[0058] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0059] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0060] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for anthropomorphic interaction of intelligent agents based on multimodal cognitive graphs and large model middleware, characterized in that, The method includes: In response to user-triggered interaction requests, collect the user's current interaction data; The current interaction data is input into a pre-constructed multimodal cognitive graph for context analysis, which identifies the user's explicit intentions and implicit emotional states, and generates corresponding structured context description vectors. The structured context description vector is input into the large model middleware, which calls a pre-set anthropomorphic strategy template to generate initial response content that matches the explicit intent and the implicit emotional state. The initial response content is decomposed into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters through the parameterized adaptation layer built into the large model middleware. Based on the speech synthesis parameters, an anthropomorphic voice is generated, and at the same time, based on the facial expression driving parameters and the action posture driving parameters, a preset three-dimensional virtual image is driven to make coordinated expressions and actions, forming a multimodal synchronous output; During and after the multimodal synchronous output, user feedback interaction data is continuously collected. The confidence level of the feedback interaction data is evaluated through the multimodal cognitive map to generate an interaction deviation index. By integrating the structured context description vector, the initial response content, and the interaction deviation index, the weights of the intent-response path in the multimodal cognitive graph are dynamically updated using a reinforcement learning algorithm, and a human-like quality score for this interaction is output.

2. The method according to claim 1, characterized in that, The step of inputting the current interaction data into a pre-constructed multimodal cognitive graph for contextual analysis to identify the user's explicit intentions and implicit emotional states includes: The current interactive data is decomposed into three types of modal nodes: speech features, image features, and text features; Cross-modal link retrieval is performed on the multimodal cognitive graph using the three types of modal nodes; If a matching path is found that is directly related to the three types of modal nodes, the target intent node at the end of the matching path is taken as the explicit intent, and the sentiment tag associated with the target intent node is taken as the implicit sentiment state. If no direct matching path is found, the graph inference engine is activated, and the association probability with each candidate intent node is calculated through the graph attention network. Candidate intents with association probabilities exceeding a preset threshold are selected as the explicit intents, and the corresponding sentiment tags are output synchronously.

3. The method according to claim 1, characterized in that, The large model middleware calls a pre-set anthropomorphic strategy template to generate initial response content that matches the explicit intention and the implicit emotional state, including: The emotional tone of the response is determined based on the implicit emotional state, and the emotional tone includes pleasure, empathy, concern, or neutrality; The functional content of the response is determined based on the explicit intent, and the functional content includes information query, task execution, emotional companionship or casual conversation; The emotional tone and the functional content are concatenated into a synthetic prompt word, which is then input into the large language model in the large model middleware to generate the initial response content.

4. The method according to claim 3, characterized in that, When inputting into the large language model in the large model middleware, the temperature parameter is set to 0.7; after generating the initial response content, the initial response content is also filtered for sensitive words and its length is pruned to ensure that it meets the preset security specifications and output duration constraints.

5. The method according to claim 1, characterized in that, The decomposition of the initial response content into speech synthesis parameters, facial expression driving parameters, and motion posture driving parameters through the parameterized adaptation layer built into the large model middleware includes: The text of the initial response is input into the text encoder of the parameterized adaptation layer to obtain the text embedding vector; The text embedding vector is mapped to the speech synthesis parameter space, the facial blending shape parameter space, and the skeletal animation parameter space respectively to output the speech synthesis parameters, the facial expression driving parameters, and the motion posture driving parameters. The speech synthesis parameters include phoneme duration, fundamental frequency curve, and spectral envelope; the facial expression driving parameters include mouth opening and closing degree, eyebrow raising angle, and eyelid closing degree; the action posture driving parameters include head turning angle, gesture trajectory, and torso swaying amplitude.

6. The method according to claim 5, characterized in that, The process of generating human-like speech based on the speech synthesis parameters, and simultaneously driving the 3D virtual avatar to perform coordinated expressions and movements based on the facial expression driving parameters and the motion posture driving parameters, specifically includes: Based on the phoneme duration and the spectral envelope, an anthropomorphic speech is generated by a speech synthesis engine, and the anthropomorphic speech is segmented into multiple speech segments according to the phoneme duration boundaries; During the playback of each of the aforementioned voice segments, the lip mesh vertex of the three-dimensional virtual image is displaced according to the degree of mouth opening and closing, while the eyebrow and eye movements are driven according to the eyebrow raising angle and the eyelid closing degree. The upper limbs and torso of the three-dimensional virtual image are driven according to the gesture trajectory and the torso swing amplitude, and the peak point of the gesture trajectory is aligned with the stress point of the anthropomorphic voice; the cervical skeleton of the three-dimensional virtual image is driven according to the head tilt angle. At the interval between two adjacent speech segments, or during the silence segment of the anthropomorphic speech, the gaze and blinking behavior of the three-dimensional virtual avatar are inserted, wherein the gaze is directed at the user's position, and the blinking behavior is spaced 6 to 8 seconds apart. During the playback of the anthropomorphic voice, the instantaneous slope of the fundamental frequency curve is extracted in real time; when the instantaneous slope exceeds a preset steep change threshold, the eyebrows of the three-dimensional virtual image are driven to quickly rise once and remain for 0.3 seconds before resetting to simulate a surprised or emphasized expression.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: Record the structured context description vector, the initial response content, the interaction deviation index, and the anthropomorphic quality score generated for each interaction to form an interaction history dataset; When the number of dialogue rounds for the same user in the interaction history dataset exceeds a preset round threshold, a pre-set personalized fine-tuning model is invoked. Based on the interaction history dataset, the node weights in the multimodal cognitive graph and the low-rank adaptation parameters of the large model middleware are incrementally updated to generate a user-specific anthropomorphic interaction configuration.

8. A human-like interactive system for intelligent agents based on multimodal cognitive graphs and large model middleware, characterized in that, include: The data acquisition module is used to collect the user's current interaction data in response to the user's interactive request. The structured context description vector generation module is used to input the current interaction data into a pre-constructed multimodal cognitive graph for context analysis, identify the user's explicit intentions and implicit emotional states, and generate corresponding structured context description vectors. The initial response content generation module is used to input the structured context description vector into the large model middleware. The large model middleware calls a preset anthropomorphic strategy template to generate initial response content that matches the explicit intention and the implicit emotional state. The initial response content decomposition module is used to decompose the initial response content into speech synthesis parameters, facial expression driving parameters, and action posture driving parameters through the parameterized adaptation layer built into the large model middleware. The multimodal synchronous output module is used to generate human-like speech based on the speech synthesis parameters, and simultaneously drive a preset three-dimensional virtual image to make coordinated expressions and movements based on the facial expression driving parameters and the action posture driving parameters, thereby forming multimodal synchronous output. The interaction deviation index generation module is used to continuously collect user feedback interaction data during and after the multimodal synchronous output, evaluate the confidence level of the feedback interaction data through the multimodal cognitive map, and generate an interaction deviation index. The anthropomorphic quality score output module is used to integrate the structured context description vector, the initial response content, and the interaction deviation index, dynamically update the weights of the intent-response path in the multimodal cognitive graph through a reinforcement learning algorithm, and output the anthropomorphic quality score of this interaction.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images