Humanoid agent cognitive behavior integrated system with cooperation of large and small models

Abstract

The invention, which relates to the technical field of humanoid agent interaction, discloses a humanoid agent cognitive behavior integrated system with cooperation of large and small models, comprising a data identification unit, an intention understanding unit, a memory storage unit, a question and answer reasoning unit and an expression output unit. The core function of a human-like agent is completed in a large and small model cooperation mode, the small model integrates vertical domain data to endow the agent with functions of intention understanding, visual perception and behavior and emotion deduction in the vertical domain, the large model is responsible for reasoning of complex semantics, and the system constructs each function module in a large and small model mode, so that the system is more intelligent. A human-like agent framework capable of real-time interaction is formed, pre-analysis is performed on an open intention of a user before deep reasoning is performed on an end-to-end voice large model, and the mechanism is specially used for solving field specific instructions or professional knowledge which are difficult to understand in the vertical field of the voice large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-like agent interaction, in particular to a human-like agent cognitive behavior integration system based on large and small models. BACKGROUND

[0002] A human-like agent cognitive behavior integration system based on large and small models, in which a large model is responsible for high-level reasoning, knowledge integration and planning, and a small model handles real-time perception, rapid response and local decision-making, forming complementary cooperation, simulating human thinking and behavior patterns, and having both versatility and efficiency. It can be applied to robots, virtual assistants and other scenarios. In the patent application No. 202411038075.9, a distributed agent system and method for realizing human-like agents based on large language models are disclosed, which relates to the technical field of human-like agents. The system includes an intelligent ontology and a behavior library. The intelligent ontology is composed of an intelligent core body, an environment perception module, a vector database, a planning module and an immediate behavior module. The present application improves stability and reduces illusion effect. Under the support of the cell agent architecture, specific safety feedback cell agents, memory retrieval cell agents and network knowledge cell agents are constructed. The safety feedback cell agent detects the rationality and feasibility of the generated content of the intelligent agent. If the detection fails, the guided intelligent agent is improved for further reflection and regeneration. The memory retrieval and network knowledge cell agents respectively ensure that the illusion effect of the generated content of the intelligent agent is minimized from the private memory library of the intelligent agent and network search, and can update the information in real time.

[0003] The above-mentioned prior art solves the problem of instability in long-time operation or processing complex tasks, but the system relies entirely on large language models to process text-level semantic understanding and content generation during operation, resulting in a long processing flow and relatively slow overall response speed. In addition, the current human-like agent is limited to voice and text question and answer based on instructions, and lacks understanding ability, emotional action expression ability and vertical domain cognitive ability in complex scenarios. SUMMARY

[0004] The present application aims to provide a human-like agent cognitive behavior integration system based on large and small models to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a human-like agent cognitive behavior integration system based on large and small models,

[0006] A data recognition unit collects text information, audio signals and image information through a signal acquisition device, extracts image features, converts the feature sequence into a visual description text, and converts the audio signal into corresponding text information.

[0007] an intention understanding unit configured to obtain text semantic representations and semantic information, analyze the text semantic representations and the semantic information using natural language processing techniques, and obtain user intention information;

[0008] a memory storage unit configured to store various types of information using a hierarchical architecture, divide the information into two categories, namely short-term memory and long-term memory, construct an enhanced context according to current semantic information and user intention information, and modify and supplement existing long and short-term memory content;

[0009] a question and answer reasoning unit configured to construct an action generation model and an emotion calculation model through reinforcement learning techniques, generate a semantic content sequence of a reply using a voice large model, and call the action generation model and the emotion calculation model to output a body action sequence and a facial expression sequence;

[0010] an expression output unit configured to obtain the voice content sequence, the body action sequence, and the facial expression sequence, control a head display screen of the agent to play a dynamic expression image, and drive a movable facial component to express an emotional tendency through light color changes, and simultaneously play the voice content sequence through a built-in loudspeaker of the agent.

[0011] Preferably, the data recognition unit comprises a data acquisition module configured to acquire external information of the agent through a signal acquisition device, wherein the external information includes text information, audio signals, and image information, the audio signals include but are not limited to human voice dialog, environmental noise, and specific sound events, and the image information includes but is not limited to scene pictures, character pictures, space layouts, and text identifiers.

[0012] Preferably, the data recognition unit further comprises an image data analysis module and a voice signal conversion module, wherein the image data analysis module uses a convolutional neural network technique to construct a visual recognition model, extracts image features using the visual recognition model, converts the feature sequence into a visual description text after identifying key elements in the image, and the voice signal conversion module uses a speech recognition technique to construct a voice analysis model, and converts the audio signal into corresponding text information using the voice analysis model.

[0013] Preferably, the types of text information in the data acquisition module include:

[0014] a. Agent character background and personality portrait, specifically, basic setting texts defining the identity, social role, core personality characteristics, behavior tendency, and value orientation of the agent;

[0015] b. The pre-defined memory content of the agent, specifically the structured text data of the prior knowledge, historical events, user preferences, key facts and vertical domain expertise that the agent should have in the initial state or specific scenarios, constitutes the basic knowledge base;

[0016] c. Scene information, specifically the context description text that provides the current environment or task, including scene setting, task target description, related rule explanation, vertical domain expertise and background information of other roles.

[0017] Preferably, the intent understanding unit includes a speech information extraction module and an intent information extraction module. The speech information extraction module forms a unified text semantic representation using visual description text and text information to extract basic semantic information. The intent information extraction module uses natural language processing technology to analyze the input text semantic representation to obtain user intent information and assess emotional tendencies.

[0018] Preferably, the memory storage unit includes a memory division module and a memory update module. The memory division module uses a hierarchical architecture to store various types of information, dividing the information into two categories: short-term memory and long-term memory. The short-term memory contains immediate information for the current interaction, while the long-term memory contains knowledge and experience information. The memory update module retrieves historical dialogues, role settings and scene information from the core storage items based on current semantic information and user intent information to build an enhanced context. It also incorporates new interactive information into storage in real time, while modifying and supplementing existing short-term and long-term memory content. According to pre-set rules, the information is filtered, simplified and archived.

[0019] Preferably, the core storage items in the memory update module include the agent's own role setting and background information, the environmental scene context information of the interaction, the complete dialogue history information, the user behavior history information and the agent's own behavior history information.

[0020] Preferably, the question and answer reasoning unit comprises a model construction module, a voice reply module and a sequence deduction module, the model construction module constructs an action generation model and an emotion calculation model through reinforcement learning technology, the voice reply module determines that the intelligent agent needs to perform voice reply if the enhanced context contains a question and answer mark, determines the performance priority, and selects an execution scheme according to the priority, a first execution scheme adopts a large language model to process semantic understanding and content generation at the text level, outputs a high-quality text reply, and then converts the text reply into a playable semantic content sequence, a second execution scheme adopts an end-to-end voice large model to generate a reply semantic content sequence and an implied emotional tendency according to the context of the current dialogue, and calls the action generation model and the emotion calculation model to cooperatively generate a body action sequence and a facial expression sequence highly matched with the semantic content sequence, the action generation model is internally provided with a behavior conflict detection and arbitration mechanism, which strictly examines all to-be-executed requests before finally sending an execution instruction to the driving layer, makes real-time decisions by using preset priority rules, suppresses low-priority and mutually conflicting requests, and only allows the behavior instruction with the highest priority and logical consistency to pass and execute, and the sequence deduction module determines that the intelligent agent does not need to perform voice reply if the enhanced context does not contain a question and answer mark, in a non-question and answer scene, the action generation model and the emotion calculation model deduce the intelligent agent body action sequence and facial expression sequence at the next time point according to the current state.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] The present application completes the expression function of the humanoid intelligent agent in the form of cooperation between large and small models, the small model integrates vertical domain data to give the intelligent agent the intention understanding, visual perception, behavior and emotion deduction function in the vertical field, and the large model is responsible for complex semantic reasoning, the system constructs various functional modules in the form of large and small models to form a real-time interactive humanoid agent framework, and performs pre-analysis on the user's intention before deep reasoning by the end-to-end voice large model, this mechanism is specially designed to solve the difficulty of the voice large model in understanding the field-specific instructions or professional knowledge in the vertical field, and at the same time, the action generation model and the emotion calculation model are established by quickly fitting the vertical role behavior data to ensure that the body action and emotional expression of the intelligent agent can be more rich. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 An overall system flowchart is provided for the embodiments of the present application;

[0024] Figure 2 An internal module block diagram of the data recognition unit provided for the embodiments of the present application is provided;

[0025] Figure 3An internal module block diagram of the memory storage unit provided for the embodiment of the present application is provided.

[0026] Figure 4 An overall internal module block diagram provided for the embodiment of the present application is provided.

[0027] In the figure: 1, a data recognition unit; 101, a data acquisition module; 102, an image data analysis module; 103, a voice signal conversion module; 2, an intention understanding unit; 201, a voice information extraction module; 202, an intention information extraction module; 3, a memory storage unit; 301, a memory division module; 302, a memory update module; 4, a question and answer reasoning unit; 401, a model construction module; 402, a voice reply module; 403, a sequence deduction module; 5, an expression output unit. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0029] Embodiment:

[0030] Please refer to Figures 1-4 The present application provides a technical solution: a human-like intelligent agent cognitive behavior integrated system with large and small models cooperating.

[0031] The data recognition unit 1 collects text information, audio signals and image information through a signal acquisition device, converts the feature sequence into a visual description text after extracting image features, and converts the audio signals into corresponding text information.

[0032] The intention understanding unit 2 obtains text semantic representation and semantic information, analyzes it using natural language processing technology, and obtains user intention information.

[0033] The memory storage unit 3 stores various types of information using a hierarchical architecture, divides the information into two categories, namely short-term memory and long-term memory, constructs an enhanced context according to the current semantic information and user intention information, and corrects and supplements the existing long and short-term memory content.

[0034] The question and answer reasoning unit 4 constructs an action generation model and an emotion calculation model through reinforcement learning technology, generates a semantic content sequence of the reply using a voice large model, and calls the action generation model and the emotion calculation model to output a sequence of body movements and a sequence of facial expressions.

[0035] The expression output unit 5 acquires the voice content sequence, the body movement sequence and the facial expression sequence, controls the agent head display screen to play the dynamic expression image, drives the movable facial component to express the emotional tendency through the light color change, and synchronously plays the voice content sequence through the built-in speaker of the agent.

[0036] The data recognition unit 1 includes a data collection module 101, which collects external information of the agent through a signal collection device. The external information includes text information, audio signals and image information. The audio signals include but are not limited to human voice dialogue, environmental noise and specific sound events. The image information includes but is not limited to scene pictures, character pictures, space layouts and text labels.

[0037] The data recognition unit 1 further includes an image data analysis module 102 and a voice signal conversion module 103. The image data analysis module 102 uses a convolutional neural network technology to construct a visual recognition model, extracts image features using the visual recognition model, and converts the feature sequence into a visual description text after identifying the key elements in the image. The voice signal conversion module 103 uses a speech recognition technology to construct a voice analysis model, and converts the audio signal into corresponding text information using the voice analysis model.

[0038] The types of text information in the data collection module 101 include:

[0039] a. Agent character background and personality portrait, specifically the basic setting text defining the identity, social role, core personality characteristics, behavior tendency and value orientation of the agent;

[0040] b. Predefined memory content of the agent, specifically structured text data of prior knowledge, historical events, user preferences, key facts and vertical professional knowledge that the agent should have in the initial state or specific scenarios, constituting a basic knowledge base;

[0041] c. Scene information, specifically context description text providing the current environment or task, including scene setting, task target description, related rule explanation, vertical domain professional knowledge and background information of other characters;

[0042] The intent understanding unit 2 includes a voice information extraction module 201 and an intent information extraction module 202. The voice information extraction module 201 forms a unified text semantic representation using the visual description text and the text information, thereby extracting basic semantic information. The intent information extraction module 202 analyzes the input text semantic representation using natural language processing technology to obtain user intent information and assess emotional tendency.

[0043] The memory storage unit 3 comprises a memory division module 301 and a memory update module 302. The memory division module 301 adopts a hierarchical architecture to store various types of information, and divides the information into two categories, namely short-term memory and long-term memory. The short-term memory contains instant information of the current interaction, and the long-term memory contains knowledge and experience information. The memory update module 302 retrieves historical dialogues, role settings and scene information from the core storage items according to the current semantic information and user intent information to construct an enhanced context, and simultaneously incorporates the newly generated interaction information into the storage, corrects and supplements the existing long-term and short-term memory content, and filters, simplifies and archives the information according to the preset rules.

[0044] The core storage items in the memory update module 302 include the role setting and background information of the agent itself, the environmental scene context information of the interaction, the complete dialogue history information, the user behavior history information and the agent behavior history information.

[0045] The question and answer reasoning unit 4 comprises a model construction module 401, a voice reply module 402 and a sequence deduction module 403. The model construction module 401 constructs an action generation model and an emotion calculation model through reinforcement learning technology. If the enhanced context contains a question and answer flag, the voice reply module 402 determines that the agent needs to perform voice reply, determines the performance priority, and selects the execution scheme according to the priority. The first execution scheme adopts a large language model to process semantic understanding and content generation at the text level, outputs high-quality text reply, and then converts the text reply into a playable semantic content sequence. The second execution scheme adopts an end-to-end voice large model to generate a reply semantic content sequence and an implied emotional tendency according to the current dialogue context, and calls the action generation model and the emotion calculation model to cooperatively generate a body action sequence and a facial expression sequence highly matched with the semantic content sequence. The action generation model has a built-in behavior conflict detection and arbitration mechanism, which strictly examines all pending requests before sending the execution instruction to the driving layer, makes real-time decisions using preset priority rules, suppresses low-priority and mutually conflicting requests, and only allows the highest-priority and logically consistent behavior instructions to pass and execute. If the enhanced context does not contain a question and answer flag, the sequence deduction module 403 determines that the agent does not need to perform voice reply. In a non-question and answer scenario, the action generation model and the emotion calculation model deduce the agent body action sequence and facial expression sequence at the next time point according to the current state.

[0046] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0047] While the embodiments of the application have been shown and described herein, it is to be understood that the scope of the application, jointly pointed out in the appended claims, is not to be limited to the above-described embodiments but can be otherwise variously changed, modified, replaced, and altered within the principles and spirit of the present application.

Claims

1. A humanoid intelligent agent cognitive behavior integrated system with large and small model collaboration, characterized in that: The data recognition unit (1) collects text information, audio signals and image information through a signal acquisition device, extracts image features, converts the feature sequence into visual descriptive text, and converts audio signals into corresponding text information. The intent understanding unit (2) acquires text semantic representation and semantic information, analyzes them using natural language processing technology, and obtains user intent information. The memory storage unit (3) adopts a hierarchical architecture to store various types of information, dividing the information into two categories: short-term memory and long-term memory. It constructs an enhanced context based on the current semantic information and user intent information, and corrects and supplements the existing long and short-term memory content. Question answering reasoning unit (4), the question answering reasoning unit (4) constructs an action generation model and an emotion computing model through reinforcement learning technology, uses a large speech model to generate a semantic content sequence of the reply, and calls the action generation model and the emotion computing model to output a body action sequence and a facial expression sequence; The expression output unit (5) acquires the speech content sequence, body movement sequence and facial expression sequence, controls the intelligent agent's head display screen to play dynamic expression images, drives movable facial parts, expresses emotional tendencies through changes in light color, and simultaneously plays the speech content sequence through the intelligent agent's built-in speaker.

2. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 1, characterized in that: The data recognition unit (1) includes a data acquisition module (101). The data acquisition module (101) collects external information of the intelligent agent through a signal acquisition device. The external information includes text information, audio signals and image information. The audio signals include, but are not limited to, human voice dialogue, environmental noise and specific sound events. The image information includes, but is not limited to, scene images, character images, spatial layout and text labels.

3. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 2, characterized in that: The data recognition unit (1) further includes an image data analysis module (102) and a speech signal conversion module (103). The image data analysis module (102) uses convolutional neural network technology to construct a visual recognition model, extracts image features using the visual recognition model, identifies key elements in the image, and converts the feature sequence into visual descriptive text. The speech signal conversion module (103) uses speech recognition technology to construct a speech analysis model, and uses the speech analysis model to convert audio signals into corresponding text information.

4. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 2, characterized in that: The types of text information in the data acquisition module (101) include: a. Background and personality profile of the intelligent agent, specifically the basic setting text that defines the identity, social role, core personality traits, behavioral tendencies and value orientation of the intelligent agent; b. The predefined memory content of the intelligent agent is specifically loaded as structured text data of prior knowledge, historical events, user preferences, key facts and domain-specific professional knowledge that the intelligent agent should possess in the initial state or specific scenario, forming a basic knowledge base; c. Scenario information, specifically providing contextual descriptions of the current environment or task, including scenario settings, task objective descriptions, relevant rule explanations, vertical domain expertise, and background information of other roles.

5. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 1, characterized in that: The intent understanding unit (2) includes a voice information extraction module (201) and an intent information extraction module (202). The voice information extraction module (201) uses visual description text and text information to form a unified text semantic representation, thereby extracting basic semantic information. The intent information extraction module (202) uses natural language processing technology to analyze the input text semantic representation, obtain user intent information, and evaluate emotional tendencies.

6. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 1, characterized in that: The memory storage unit (3) includes a memory partitioning module (301) and a memory update module (302). The memory partitioning module (301) uses a hierarchical architecture to store various types of information, dividing the information into two categories: short-term memory and long-term memory. The short-term memory contains the immediate information of the current interaction, while the long-term memory contains knowledge and experience information. The memory update module (302) retrieves historical dialogues, role settings, and scene information from the core storage items based on the current semantic information and user intent information to construct an enhanced context. It also incorporates newly generated interaction information into the storage in real time, while correcting and supplementing the existing long and short-term memory content, and filtering, simplifying, and archiving the information according to preset rules.

7. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 6, characterized in that: The core storage items in the memory update module (302) include the agent's own role setting and background information, the environmental context information of the interaction, the complete dialogue history information, the user behavior history information, and the agent's own behavior history information.

8. The integrated cognitive and behavioral system for humanoid intelligent agents with large and small model collaboration according to claim 1, characterized in that: The question-and-answer reasoning unit (4) includes a model building module (401), a voice response module (402), and a sequence deduction module (403). The model building module (401) constructs an action generation model and an emotion computing model through reinforcement learning technology. If the enhanced context contains question-and-answer markers, the voice response module (402) determines that the agent needs to provide a voice response, determines the performance priority, and selects an execution plan according to the priority. The first execution plan uses a large language model to process semantic understanding and content generation at the text level, outputs a high-quality text response, and then converts the text response into a more accessible format. The second execution scheme uses an end-to-end speech model to generate a semantic content sequence of the response and the implied emotional tendency based on the context of the current dialogue. It also calls the action generation model and the emotion computing model to collaboratively generate a body action sequence and facial expression sequence that are highly matched with the semantic content sequence. If the enhanced context does not contain a question-and-answer flag, the sequence inference module (403) determines that the agent does not need to make a voice response. In non-question-and-answer scenarios, the action generation model and the emotion computing model will infer the agent's body action sequence and facial expression sequence for the next time point based on the current state.

Images