Multi-modal intelligent humanization interaction method and system for multi-end long memory

Abstract

The application discloses a multi-modal intelligent personification interaction method and system for multi-end long memory, relates to the technical field of intelligent conversation, and comprises the following steps: performing memory retrieval on a target conversation request by using a preset memory retrieval mechanism, obtaining a target memory segment from a memory storage area, and performing enhancement processing on the target memory segment by using a preset knowledge graph to obtain an enhanced memory segment; constructing a user portrait vector according to user basic information and historical memory, filling a preset personality instruction template, and generating a target scenario-based personality instruction; based on the enhanced memory segment, the target scenario-based personality instruction, the emotional state of the target conversation request, and the request intention identified by the target conversation request; adjusting the mode weight of each output mode according to the emotion vector mapped by the emotional state, calling a reply generation model to generate reply content according to the request intention and the mode weight, and returning after security audit.

Description

Technical Field

[0001] This invention relates to the field of intelligent dialogue technology, and in particular to a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method and system. Background Technology

[0002] Current conversational interaction methods typically rely on rules or simple classification models to determine the type of user request, thereby identifying the corresponding request intent. Based on the intent, the request is routed to the appropriate service interface, which independently generates the response content and returns it to the front end for display. If the user initiates further conversation based on the same request, the lack of memory and association with past conversation content may lead to a re-generation of the response content instead of modification. If the user logs in again after a few days, the conversation will start from scratch. Because existing technologies rely on short-term memory based on conversations or are limited to the context window of the LLM (Large Language Model), historical information is lost once the conversation ends or the number of conversation turns exceeds the window limit, resulting in a disjointed and repetitive conversational experience and a poor user experience. Furthermore, treating different modalities as independent, passively triggered functional modules and using keyword-based hard distribution for intent recognition leads to rigid and fragmented interaction methods. Users must use precise commands that conform to the system's preset logic, resulting in an unnatural experience. Furthermore, the response style of existing technologies is usually fixed, determined by the general pre-training characteristics of the underlying LLM or simple system-level instructions. As a result, the presentation style of the system is always the same, leading to a mechanical dialogue experience that lacks emotional resonance and personalization.

[0003] In summary, how to solve the problems of short memory, single interaction modality, and mechanical response content that are common in existing conversational interaction methods is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multi-modal intelligent anthropomorphic interaction method and system with long-term memory across multiple devices, which can solve the problems of short-term memory, single interaction modality, and mechanical response content that are common in existing conversational interaction methods. The specific solution is as follows:

[0005] Firstly, this application provides a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method, applied to a multimodal intelligent anthropomorphic interaction system, comprising:

[0006] A preset memory retrieval mechanism is used to retrieve the target session request initiated by the target client to obtain the relevant target memory fragment from the memory storage area. The target memory fragment is then enhanced using a preset knowledge graph to obtain the enhanced memory fragment.

[0007] Based on the user's basic information and historical memory corresponding to the target session request, a corresponding user profile vector is constructed, and the user profile vector is used to fill the preset personality instruction template to generate the target contextualized personality instruction; the user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags;

[0008] Based on the enhanced post-memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, the request intent is identified to obtain the corresponding request intent;

[0009] The modality weights of each output modality are adjusted according to the emotion vector of the emotion state mapping, so as to call the corresponding response generation model according to the request intent and the modality weights, generate the response content of the target session request, and return it after the response content passes the security review.

[0010] Optionally, the memory storage area includes a first memory storage area and a second memory storage area; the first memory storage area is a preset memory cache area; the second memory storage area is a preset vector database for storing memory fragments.

[0011] Optionally, the step of using a preset memory retrieval mechanism to perform memory retrieval on a target session request initiated by a target client, in order to obtain relevant target memory fragments from the memory storage area, includes:

[0012] Based on the user identifier corresponding to the target session request, a search is performed in the first memory storage area to determine candidate memory segments with the user identifier as the key;

[0013] The target session request is encoded into a target query vector using a preset vector transformation model, and the first cosine similarity between the candidate memory fragment and the target query vector is determined.

[0014] When the first cosine similarity satisfies the preset vector retrieval conditions, the target historical memory fragment is selected from the second memory storage area based on the second cosine similarity between the historical memory fragment and the target query vector. When the preset mixed retrieval conditions are met, keyword full-text retrieval is performed based on the target session request to obtain the corresponding retrieval results.

[0015] The search results and the target historical memory fragments are reordered based on the target semantic association to determine the relevant target memory fragments; the target semantic association refers to the semantic association between the target session request and the search results and the target historical memory fragments, respectively.

[0016] Optionally, several memory segments in the memory storage area have corresponding metadata information; the metadata information includes a storage timestamp, a last access timestamp, and a memory segment weight; the lifetime of several cached memory segments in the first memory storage area is positively correlated with the user's subscription level; the upper limit of the number of memory segments stored in the second memory storage area is positively correlated with the user's subscription level; the upper limit of the memory retrieval frequency of the memory storage area within a preset time period is positively correlated with the user's subscription level;

[0017] Accordingly, after the response content passes the security audit and is returned, the process also includes:

[0018] If there is a memory fragment to be strengthened in the enhanced memory fragment that has a positive feedback effect on the current dialogue interaction, then the weight of the memory fragment to be strengthened is increased according to a preset logarithmic function with time decay.

[0019] A number of memory segments in the memory storage area are scanned based on a preset memory scanning frequency. The weight of the memory segment to be decayed among the memory segments is reduced based on a preset decay factor. When a memory segment to be eliminated is detected in the memory storage area, the memory segment to be eliminated is eliminated. The memory segment to be decayed is the memory segment whose difference between the last access timestamp and the current timestamp is greater than a preset difference threshold. The memory segment to be eliminated is the memory segment whose weight is lower than a preset memory segment weight threshold.

[0020] The preset difference threshold is positively correlated with the user subscription level; the preset memory fragment weight threshold is negatively correlated with the user subscription level.

[0021] Optionally, the step of enhancing the target memory fragment using a preset knowledge graph to obtain an enhanced memory fragment includes:

[0022] Identify the target entity in the target session request, and use a preset knowledge graph to match the predefined entity relationships of the target entity;

[0023] Identify the relevant memory fragments in the target memory fragments that are related to the target entity, and associate and bind the relevant entities in the relevant memory fragments that correspond to the target entity with the preset knowledge graph, so as to extract the entity association information between the target entities based on the obtained binding relationship and the predefined entity relationship;

[0024] The entity association information is appended to the target memory fragment as extended semantic information to obtain the enhanced memory fragment.

[0025] Optionally, the step of constructing a corresponding user profile vector based on the user's basic information and historical memory corresponding to the target session request, and using the user profile vector to fill in a preset personality instruction template to generate a target contextualized personality instruction, includes:

[0026] Determine the user basic information corresponding to the target session request, and extract the corresponding structured data from the user basic information to generate corresponding basic attribute tags based on the structured data and the location of the target client;

[0027] Statistical analysis of user interaction behavior data in historical memory is used to generate behavioral preference labels based on the analysis results;

[0028] A preset topic modeling algorithm is used to perform topic clustering on the dialogue content in the user's historical memory, and topics are supplemented based on the user's historical search information to obtain corresponding topic preference tags;

[0029] A preset sentiment classification model is used to perform sentiment analysis on the user's historical memories and corresponding historical feedback behaviors to generate corresponding sentiment feature labels.

[0030] Construct a corresponding user profile vector based on the basic attribute tags, the behavioral preference tags, the topic preference tags, and the emotional feature tags;

[0031] By using a preset dynamic content filling engine, the preset personality instruction template is filled with the user profile vector and the emotional state to generate target contextualized personality instructions.

[0032] Optionally, adjusting the modal weights of each output modality based on the emotion vector mapped from the emotion state includes:

[0033] The emotional state is evaluated using a pre-defined multimodal emotion classification model, and the emotional state is mapped into an emotion vector based on the evaluation results.

[0034] If the emotional valence component in the emotional vector is lower than a preset emotional valence component threshold, then the modal weight of the non-text output modality is increased; the non-text output modality includes the image output modality and the audio output modality.

[0035] If the emotional valence component in the emotional vector is not lower than the preset emotional valence component threshold, then the modal weight of the text output modality is controlled to be higher than the modal weight of the non-text output modality.

[0036] Optionally, returning the response after the content has passed security review includes:

[0037] The response content is subjected to security review using a local preset sensitive word recognition algorithm and a content review service called through a preset security review interface;

[0038] If the security audit operation fails, the corresponding user group type is determined based on the user profile vector, and the risk control level of the current interaction is determined. Based on the risk control level and the user group type, corresponding security prompts and feedback are given to the target session request.

[0039] If the security audit has been approved, the response content will be formatted and returned.

[0040] Optionally, after the response content passes security review and is returned, the method further includes:

[0041] The system acquires response behaviors to the given answers and analyzes these behaviors based on the target contextualized personality instructions. It then determines the feedback type of the response behavior based on the analysis results and adjusts the weights of relevant tags in the user profile vector according to the feedback type. The feedback types include positive feedback and negative feedback.

[0042] Alternatively, based on a preset vector update frequency, the label weights in the user profile vector can be adjusted using user interaction data within a preset historical time period.

[0043] Secondly, this application provides a multi-terminal, long-memory, multimodal intelligent anthropomorphic interactive system, including:

[0044] The memory retrieval module is used to perform memory retrieval on the target session request initiated by the target client using a preset memory retrieval mechanism, so as to obtain the relevant target memory fragments from the memory storage area, and to enhance the target memory fragments using a preset knowledge graph to obtain enhanced memory fragments.

[0045] The template filling module is used to construct a corresponding user profile vector based on the user's basic information and historical memory corresponding to the target session request, and to fill the preset personality instruction template with the user profile vector to generate the target contextualized personality instruction; the user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags and topic preference tags;

[0046] The intent recognition module is used to identify the request intent based on the enhanced post-memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, so as to obtain the corresponding request intent;

[0047] The weight adjustment module is used to adjust the modal weights of each output modality according to the emotion vector mapped by the emotion state, so as to call the corresponding response generation model according to the request intent and the modal weights, generate the response content of the target session request, and return the response content after it passes the security review.

[0048] In this application, a preset memory retrieval mechanism is used to retrieve the target session request initiated by the target client, thereby obtaining relevant target memory fragments from the memory storage area. These memory fragments are then enhanced using a preset knowledge graph to obtain enhanced memory fragments. A corresponding user profile vector is constructed based on the user's basic information and historical memory corresponding to the target session request. This user profile vector is then used to fill a preset personality instruction template, generating a target contextualized personality instruction. The user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags. Request intent is identified based on the enhanced memory fragments, the target contextualized personality instruction, the emotional state of the target session request, and the target session request itself, yielding the corresponding request intent. The modal weights of each output modality are adjusted according to the emotion vector mapped from the emotional state. Based on the request intent and the modal weights, a corresponding response generation model is invoked to generate the response content for the target session request. This response content is then returned after passing security verification. As can be seen from the above, the multimodal intelligent anthropomorphic interactive system of this application performs memory retrieval on the target session request initiated by the target client through a preset memory retrieval mechanism, extracts the relevant target memory fragments from the memory storage area, and obtains enhanced memory fragments through preset knowledge graph enhancement. Combined with the user's basic information and user's historical memory corresponding to the target session request, a user profile vector including basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags is constructed. This vector is used to fill a preset personality instruction template to generate a target contextualized personality instruction. Based on the enhanced memory fragments, the target contextualized personality instruction, the emotional state of the target session request, and the request itself, intent recognition is performed to obtain the request intent. The weights of each output modality are adjusted according to the emotion vector mapped by the emotional state. Then, the response generation model is called to generate response content according to the request intent and the modality weights. After security review, the response is returned.In this way, through the process described above in this application, relevant target memory fragments are extracted through a memory retrieval mechanism. These fragments are then combined with user basic information and historical memory to construct a user profile vector and generate contextualized personality instructions. This allows conversational interaction to fully integrate past user information and personalized traits, making responses more aligned with user habits and needs. After obtaining the target memory fragments, a pre-defined knowledge graph is used to enhance them, achieving semantic expansion and entity association. This solves the problem of information fragmentation that may exist in the original memory fragments and endows the system with cross-conversation reasoning capabilities, avoiding users repeatedly explaining their needs and further optimizing the coherence and professionalism of long-memory conversations. Based on multi-dimensional information such as emotional state and memory fragments, the system identifies request intent, significantly improving the accuracy of intent judgment compared to identifying only the request content, and reducing misunderstandings caused by incomplete information. The emotional state is transformed into an emotion vector, and the output modality weights are adjusted, enabling responses to match the user's current emotions and enhancing the emotional resonance of the interaction. After the response is generated, it undergoes security review before being returned, ensuring the security of the interactive content and preventing the transmission of harmful information. This addresses the common problems of short memory span, single interaction modality, and mechanical response content in existing conversational interaction methods. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a flowchart of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method disclosed in this application;

[0051] Figure 2 This is a system architecture diagram of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method disclosed in this application;

[0052] Figure 3 This is a schematic diagram illustrating the specific process of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method disclosed in this application;

[0053] Figure 4 This is a schematic diagram illustrating the specific process of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method disclosed in this application;

[0054] Figure 5 This is a schematic diagram of the structure of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interactive system disclosed in this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Because existing technologies rely on short-term memory based on the session or are limited to the context window of the LLM itself, historical information is lost once the session ends or the number of dialogue turns exceeds the window limit, resulting in a disjointed and repetitive dialogue experience and a poor user experience. Furthermore, treating the processing of different modalities as independent, passively triggered functional modules, and often relying on hard keyword distribution for intent recognition, leads to rigid and fragmented interaction methods. Users must use precise commands that conform to the system's preset logic, resulting in an unnatural experience. In addition, the response style of existing technologies is usually fixed, determined by the general pre-training characteristics of the underlying LLM or simple system-level commands. Therefore, the system's presentation style is always uniform, resulting in a mechanical dialogue experience lacking emotional resonance and personalization.

[0057] To overcome the aforementioned technical problems, this application provides a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method that can solve the problems of short memory, single interaction modality, and mechanical response content that are common in existing conversational interaction methods.

[0058] See Figure 1 As shown, this embodiment of the invention discloses a multi-terminal long-memory multimodal intelligent anthropomorphic interaction method, applied to a multimodal intelligent anthropomorphic interaction system, including:

[0059] Step S11: Use a preset memory retrieval mechanism to perform memory retrieval on the target session request initiated by the target client, so as to obtain the relevant target memory fragments from the memory storage area, and use a preset knowledge graph to enhance the target memory fragments to obtain enhanced memory fragments.

[0060] In this embodiment, the multimodal intelligent anthropomorphic interactive system receives a target session request initiated by a user on a target client through a target interaction interface and then standardizes it. If it is voice, the system calls the ASR (Automatic Speech Recognition) service to convert it into text. If it is an image, the system extracts its URL (Uniform Resource Locator) or performs preliminary recognition. All inputs are ultimately organized into unified structured data. Subsequently, according to a preset memory retrieval mechanism, a memory retrieval operation is performed on the target session request. Target memory fragments related to the content of the target session request are selected from the memory storage area. At the same time, a preset knowledge graph is used to enhance the target memory fragments to obtain enhanced memory fragments.

[0061] It should be noted that, such as Figure 2 The diagram shows a system architecture of a multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method provided in this application. The multimodal intelligent anthropomorphic interaction system connects to multiple clients through the target interaction interface. Clients (multi-terminal, including App (Application), web page, PC (Personal Computer), mini-program, and hardware) serve as user interaction entry points, and their requests are aggregated at the server-side access layer. After unifying the request format, the access layer submits the requests to the core processing workflow for scheduling. The core processing workflow calls internal modules (input processing module, intent recognition module, memory module, anthropomorphization module, multimodal generation module, and security and review module) as needed to process requests. These modules interact with external dependent services during task execution (e.g., the memory module uses vector models and vector databases; the input processing module calls speech recognition services; the multimodal generation module calls large language models, text-to-image / image-to-image models, and speech synthesis services; the security and review module calls content review services). The final processing result is returned to the client via the core workflow and access layer. Figure 3 , Figure 4 The diagram shown is a schematic representation of the specific process of a multi-terminal long-memory multimodal intelligent anthropomorphic interaction method provided in this application.

[0062] It should be noted that the processing flow for retrieving relevant target memory fragments by using a preset memory retrieval mechanism for a target session request is as follows: A search is performed in the first memory storage area based on the user identifier corresponding to the target session request to determine candidate memory fragments with the user identifier as the key; the target session request is encoded into a target query vector using a preset vector transformation model, and a first cosine similarity is determined between the candidate memory fragments and the target query vector; when the first cosine similarity satisfies the preset vector retrieval conditions, a target historical memory fragment is selected from the second memory storage area based on the second cosine similarity between the historical memory fragment and the target query vector; and when the preset mixed retrieval conditions are met, a keyword full-text search is performed based on the target session request to obtain the corresponding search results; the search results and the target historical memory fragments are reordered according to the target semantic association to determine the relevant target memory fragments; the target semantic association refers to the semantic association between the target session request and the search results and the target historical memory fragments, respectively. The preset vector retrieval condition is that there is no target memory fragment in the candidate memory fragments whose first cosine similarity is higher than the preset cosine similarity threshold; the preset mixed retrieval condition is that there is a memory fragment in the preset number of historical memory fragments whose second cosine similarity is not higher than the preset cosine similarity threshold, or the target session request includes target keywords; the target keywords include entities and keywords that the system can recognize and that can reflect the precise matching requirements, and are dynamically determined according to the content of the user instruction.That is, firstly, the user identifier corresponding to the target session request is determined. Based on this identifier, a search is performed in the first memory storage area to identify candidate memory segments with the user identifier as the key. The target session request is then encoded into a high-dimensional target query vector in real time using a preset vector transformation model. The first cosine similarity between the candidate memory segments and the target query vector is calculated. If there is a segment in the candidate memory segments with a first cosine similarity higher than a preset cosine similarity threshold, it is returned directly. If not, a preset similarity search algorithm, such as HNSW (Hierarchical Navigable Small World) algorithm, is used to perform an efficient ANN (Approximate Nearest Neighbor) search in the second memory storage area. The search process (approximate nearest neighbor) calculates the second cosine similarity between historical memory fragments within the region and the target query vector. Based on the similarity, a preset number of target historical memory fragments most similar to the target query vector are selected. If any of the selected fragments has a second cosine similarity not higher than a threshold, or if the target session request contains target keywords, a hybrid search mode is initiated. Based on the target session request, a keyword full-text search is performed to obtain search results. Subsequently, a preset semantic analysis model is used to evaluate the target semantic association between the target session request and the search results and the target historical memory fragments, respectively. The fragments are then reordered according to the target semantic association, and finally, the preset number of most associated fragments are selected as the relevant target memory fragments.

[0063] It should be further noted that the memory storage area includes a first memory storage area and a second memory storage area; the first memory storage area is a preset memory cache area; the second memory storage area is a preset vector database for storing memory fragments. The first memory storage area can be a cache area using Redis (i.e., RemoteDictionary Server) as a high-speed caching layer and adopting a Cache-Aside mode to cache the user's short-term memory, such as interaction points within the last 5 minutes, stored in a Redis Hash or Sorted Set data structure with the user identifier (user_id) as the key; the preset vector transformation model is a specially optimized embedding model (such as Sentence-BERT or BGE); the second memory storage area can be Milvus or PostgreSQL (a relational database) integrated with pgvector (an extension plugin).

[0064] It should be noted that after obtaining the target memory fragment, this embodiment can further enhance it. The process is as follows: identify the target entity in the target session request, and use a preset knowledge graph to match the predefined entity relationship of the target entity; determine the related memory fragments in the target memory fragment that are related to the target entity, and associate and bind the related entities in the related memory fragments that correspond to the target entity with the preset knowledge graph, so as to extract the entity association information between the target entities based on the obtained binding relationship and the predefined entity relationship; and attach the entity association information as extended semantic information to the target memory fragment to obtain the enhanced memory fragment. That is, firstly, the core target entities are identified from the target session request, such as "insomnia" (symptom entity) and "sleep aid tea" (product entity). Then, with the help of a preset knowledge graph, such as a preset health management knowledge graph, the existing predefined entity relationships of the target entity are matched. For example, "insomnia" and "sleep aid method" have a "corresponding relief method" relationship (in the knowledge graph, the node "insomnia" → "corresponding relief method" edge → nodes such as "sleep aid tea", "bedtime meditation", and "melatonin supplementation"); "sleep aid tea" and "product attributes" have a "contained ingredients", "purchase channel", and "applicable population" relationship (in the knowledge graph, the node "sleep aid tea" → "contained ingredients" edge → "sour jujube seed" and "poria cocos" nodes, → "purchase channel" edge → "system store" and "offline pharmacy" nodes). Next, relevant memory fragments related to the target entity are filtered from the target memory fragments. For example, a target memory fragment related to "sleep aid tea" is: 3 days ago, a user asked, "Are there any sleep aid teas without caffeine?", and the system replied, "We recommend Brand A sleep aid tea, which contains jujube seed and poria cocos, and is caffeine-free. You can order it from the system's online store." The relevant entities corresponding to the target entity in the relevant memory fragments are then associated and bound to a preset knowledge graph. For example, "Brand A sleep aid tea" (a specific product) in the memory fragment is associated with the "Specific Brand" sub-node under the "Sleep Aid Tea" entity in the knowledge graph, and simultaneously associated with the "Caffeine-Free" attribute node and the "System Online Store" purchase channel node. Then, based on the binding relationship and predefined entity relationship, entity association information between different target entities is extracted. For example, the deep association between "sleep aid tea - insomnia" is "the jujube seed ingredient relieves insomnia by inhibiting central nervous system excitation" (in the knowledge graph, the "jujube seed" node → "pharmacological effect" edge → "inhibit central nervous system excitation" node → "corresponding symptoms" edge → "insomnia" node). Finally, the extracted entity association information is used as extended semantic information and added to the target memory fragment to form a more information-rich enhanced post-memory fragment, achieving deeper context connection and reasoning.In this way, this embodiment extracts relevant target memory fragments from the memory storage area, providing historical evidence for the system to understand user intent, reducing response deviations caused by information gaps, improving session efficiency, and enhancing the coherence and personalized experience of user interaction; prioritizing the query of the cache area, and returning directly if a match is found, achieves millisecond-level response, reduces the access pressure on the backend vector database, and ensures low latency in high-concurrency scenarios; deep retrieval is performed from the vector database by combining cosine similarity calculation, core semantic matching, and keyword full-text retrieval, balancing retrieval efficiency and coverage, capturing deep semantic relationships without missing precise matching needs dominated by specific keywords; based on the association results of the preset knowledge graph, the system's understanding of the user's current target session request is no longer limited to the surface, but achieves deeper contextual connection and reasoning, avoiding information gaps, and supporting cross-session reasoning without having to retrieve new memory fragments, achieving logical coherence across sessions.

[0065] Step S12: Construct a corresponding user profile vector based on the user's basic information and historical memory corresponding to the target session request, and use the user profile vector to fill the preset personality instruction template to generate the target contextualized personality instruction; the user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags and topic preference tags.

[0066] In this embodiment, to achieve deep adaptation of AI (artificial intelligence) to user personality, a two-layer dynamic system is constructed: the bottom layer is a continuously evolving multi-dimensional user profile, and the upper layer is contextualized personality instructions generated in real time based on this profile. First, based on the user's basic information corresponding to the target session request and combined with the user's historical memory, a user profile vector representing user characteristics is constructed. Then, this user profile vector is used as key information and filled into a preset personality instruction template. By accurately filling in the relevant placeholder items in the template, the target contextualized personality instruction is finally generated. The user profile vector includes basic attribute tags (age, gender, region), behavioral preference tags (frequently used functions, interaction habits), emotional characteristic tags (emotional stability, interaction sensitivity), and topic preference tags (professional field, interest topic). The basic attribute tags are only adjusted when the user actively modifies them or when information verification updates, making them low-frequency update tags.

[0067] Specifically, the process involves: determining the user's basic information corresponding to the target session request and extracting corresponding structured data from the user's basic information; generating corresponding basic attribute tags based on the structured data and the target client's location; statistically analyzing the user's historical interaction data to generate behavioral preference tags based on the analysis results; using a preset topic modeling algorithm to perform topic clustering on the dialogue content in the user's historical memory and supplementing the topics based on the user's historical search information to obtain corresponding topic preference tags; using a preset sentiment classification model to perform sentiment analysis on the user's historical memory and corresponding historical feedback behavior to generate corresponding sentiment feature tags; constructing corresponding user profile vectors based on the basic attribute tags, behavioral preference tags, topic preference tags, and sentiment feature tags; and using a preset dynamic content filling engine to fill a preset personality instruction template with the user profile vectors and the emotional state to generate target contextualized personality instructions.In other words, firstly, the basic user information corresponding to the target session request, i.e., user registration information, is determined. Simultaneously, relevant structured data, such as age and gender, is extracted from this basic user information. Combined with the target client's location, regional tags are added to generate corresponding basic attribute tags. Next, the user's historical interaction behavior data is statistically analyzed to generate behavioral preference tags based on the analysis results. For example, the frequency of user function calls in the past 30 days is counted; if the "text-to-image" call ratio exceeds 60%, a "frequently used function - text-to-image" tag is generated. The length of interaction statements is also counted; if short sentences (≤10 characters) account for more than 70%, a "interaction habit - short question" tag is generated. Finally, a preset topic modeling algorithm, such as LDA (Linear Discriminant Analysis), is used. Linear discriminant analysis (LDA) is used to perform topic clustering on the dialogue content in the user's historical memory. Topics are supplemented based on the user's historical search information to obtain topic preference tags. For example, if topics related to "Internet products" account for more than 40%, a "Professional Field - Internet Products" tag is generated. Combining the user's historical search keywords (such as "science fiction movie recommendations"), a "Interest Topic - Science Fiction Movies" tag is added. Then, a preset sentiment classification model is used to perform sentiment analysis on the user's historical memory and corresponding historical feedback behaviors (such as "likes," "dislikes," and "negative review texts") to generate corresponding sentiment feature tags. For example, if the last 10... If the emotional fluctuation range during the interaction is ≤20%, a "High Emotional Stability" label is generated; if the user's response rate to the adjustment of the response style exceeds 80%, a "High Interaction Sensitivity" label is generated. Then, the basic attribute labels, behavioral preference labels, topic preference labels, and emotional feature labels are integrated to construct a unified multi-dimensional user profile vector, which serves as the basis for downstream personality instruction generation. Finally, a preset dynamic content filling engine is used to fill a structured preset personality instruction template containing multiple placeholders with the user profile vector and the emotional state to generate target contextual personality instructions. For example, for a user with "Basic Attribute Label - 25-30 years old + Behavioral Preference Label - Short Questions + Emotional Feature Label - High Emotional Stability", the "Communication Style" placeholder in the template is filled with "Concise and Friendly" and the "Professionalism" placeholder is filled with "Medium (avoid oversimplification or complex expression)", generating target contextual personality instructions. In addition, this embodiment also supports differentiated response generation: synchronized tag data provides decision-making basis for the response generation process. For example, for users with the "knowledge background tag - Internet products", when generating responses related to "product requirement document writing", industry terms (such as "PRD (Product Requirements Document)" and "user stories") are automatically incorporated; for users with the "emotional characteristic tag - high interaction sensitivity", if negative emotions are detected, the proportion of emotional reassurance statements in the response is automatically increased.

[0068] It should be noted that the user profile vector is a dynamically updated data structure that integrates the user's long-term, medium-term, and instantaneous characteristics. The core dimension of the user profile vector is: static / semi-static features (StaticAttributes): basic information provided by the user (such as preference areas, language habits, etc.). Behavioral Pattern Vector: A quantitative metric extracted through feature engineering of historical dialogues, such as: Interaction Complexity: Measured using metrics like average sentence length and type-token ratio; Topic Preference Distribution: Generated by using LDA (Latent Dirichlet Allocation) or BERT-based embedding-clustering to model topics in historical dialogues and generate topic vectors representing user interests; Interaction Style: Statistically analyzing the proportion of interrogative, imperative, and exclamatory sentences in utterances to determine whether the user is exploratory, command-oriented, or emotionally expressive; Affective State Trajectory: Recording and analyzing the sequence of emotional changes in recent interactions to identify baseline emotion and emotional fluctuation patterns. It should be further noted that the following is an example of the structure of the preset personality instruction template:

[0069] SYSTEM PROMPT:

[0070] You are a personalized AI assistant. Your persona for this interaction is defined as follows:

[0071] - Communication Style: {style}

[0072] - Domain Expertise Level: {expertise_level}

[0073] - Emotional Tone: {tone}

[0074] - Proactivity Level: {proactivity}

[0075] Engage with the user based on these parameters. The user's currentemotional state is {current_emotion}.

[0076] Among them, {style} is dynamically selected based on the interaction style and complexity in the user profile, such as formal_and_detailed (for command-oriented, high-complexity users) or casual_and_encouraging (for emotionally expressive, low-complexity users); {tone} combines the emotional state trajectory in the user profile and the current real-time emotion, selecting empathetic_and_soothing or upbeat_and_energetic, etc.; {expertise_level}: adjusts the AI's expression based on the user's topic preference distribution in a specific field and the depth of historical questions, avoiding being too simplistic or obscure; {proactivity}: is determined based on the user profile's interaction style, for example, proactively providing suggestions for exploratory users, passively waiting for instructions for command-oriented users, and prioritizing emotional reassurance for emotionally expressive users. In this way, this embodiment combines user basic information and historical memory to construct user profile vectors, which can comprehensively capture the user's personalized characteristics and avoid the one-sidedness of user profiles. The dynamic content filling engine combines multi-dimensional profiles and emotional state filling templates to further improve the contextual adaptability and personalization of instructions, so that the generated target contextualized personality instructions closely match the customized content of the user's individual characteristics, thereby improving the comfort and sense of identity of user interaction.

[0077] Step S13: Based on the enhanced post-memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, perform request intent recognition to obtain the corresponding request intent.

[0078] In this embodiment, the text content, vocabulary selection, and even tone of voice (in a voice interaction scenario) input by the user in the target session request are first analyzed to determine the emotional state corresponding to the request. Then, the enhanced memory fragment, the generated target contextualized personality instruction, the emotional state, and the target session request itself are integrated. A powerful LLM or specialized classification model is used to comprehensively judge the rich context and identify the user's true request intent, such as casual conversation, text-to-image interaction, or modification based on a previous image. In this way, this embodiment first analyzes the target session request to determine the emotional state, capturing the emotional tendency of the user when initiating the request, going beyond simple semantic understanding to achieve empathetic interaction highly adapted to the user's emotional state. Integrating multiple types of key information for intent recognition significantly reduces the risk of misjudgment of intent and improves the accuracy of consciousness recognition.

[0079] Step S14: Adjust the modal weights of each output modality according to the emotion vector mapped by the emotion state, so as to call the corresponding response generation model according to the request intent and the modal weights, generate the response content of the target session request, and return the response content after it passes the security review.

[0080] In this embodiment, the emotional state is first mapped to a quantifiable emotion vector. The modality weights of each output modality are adjusted using the emotion vector. Then, the corresponding response generation model is invoked by combining the request intent and the modality weights. The response generation model generates response content for the target session request. After generating the response content, it undergoes a security review. Once the review is passed, the response content is returned to the user through the target interaction interface.

[0081] It should be noted that the process of adjusting the modal weights of each output modality based on the emotion vector mapped from the emotional state is as follows: The emotional state is evaluated using a preset multimodal emotion classification model, and the emotional state is mapped to an emotion vector based on the evaluation results. If the emotional valence component in the emotion vector is lower than a preset emotional valence threshold, the modal weight of the non-text output modality is increased. The non-text output modality includes image output modality and audio output modality. If the emotional valence component in the emotion vector is not lower than the preset emotional valence threshold, the modal weight of the text output modality is controlled to be higher than the modal weight of the non-text output modality. The emotion vector not only contains discrete emotion labels (e.g., neutral, positive, negative), but also quantifies its emotional valence and arousal. That is, firstly, the emotional state is evaluated in real time using a preset multimodal emotion classification model (such as a BERT-based text classifier combined with a speech emotion recognition model). Based on the evaluation results, it is mapped to a quantifiable emotion vector E. Then, the relationship between the emotion valence component in the emotion vector and a preset emotion valence component threshold is determined: if it is lower than the threshold, the user is judged to have negative emotional tendencies such as negativity or depression. This significantly increases the modal weights of non-text output modalities, i.e., image output modalities and audio output modalities, aiming to prioritize image generation (such as DALL-E (an AI image generator)). API (an AI drawing tool) or audio generation service provides emotional support and distraction through more expressive and emotionally engaging media (such as a comforting picture or a piece of soothing music); if the emotional valence component is not lower than the threshold, it is determined that the user has a positive or neutral emotional tendency such as positive or peaceful; the efficiency and accuracy of information transmission are prioritized, and the modal weight of the text output modality is controlled to be higher than that of the non-text output modality, that is, it is given the highest priority; the text content is mainly generated by LLM to ensure smooth dialogue and clear information.

[0082] It should be further noted that the processing flow for returning the response content after the security review is passed is as follows: The response content is subjected to a security review using a local preset sensitive word recognition algorithm and a content review service called through a preset security review interface; if the security review fails, the corresponding user group type is determined based on the user profile vector, and the risk control level of the current interaction is judged, so as to provide corresponding security prompts and feedback to the target session request based on the risk control level and the user group type; if the security review has passed, the response content is formatted and the processed response content is returned. That is, firstly, the corresponding model is invoked to generate the response content according to the different intentions of the request: if it is a question-and-answer intention, a preset large language model is invoked, and a text response is generated by combining the target prompt words that integrate memory and personality generation with the modal weights; if it is a text-to-image intention, a preset text-to-image model, such as the Stable Diffusion (a text-to-image technology) model, is invoked to extract the image description information in the target conversation request and generate an image response by combining the modal weights; if it is an image-to-image intention, a preset image-to-image model is invoked to generate a response based on the image to be modified and the modification information in the request; if it is a speech generation intention, a preset text-to-speech (TTS) model service is invoked to generate a response based on the text to be synthesized in the request. Subsequently, the response content is subjected to dual security review using a local preset sensitive word recognition algorithm and an external content review service called through a preset security review interface. The review result is obtained by combining the two. If the review fails, the user group type is determined based on the user profile vector, and the risk control level of the current interaction is determined in combination with the review result. Then, a security prompt is given to the target session request based on the risk control level and the user group type. If the review passes, the response content is formatted and finally the processed response content is returned through the target interaction interface.

[0083] It should be noted that if the response content fails the review, the interaction strategy can be dynamically adjusted. The corresponding processing flow is as follows: When the system determines that the subject is a minor based on the user profile, and the risk control module identifies that the interaction content involves topics related to moderate violence, a guided persuasion strategy will be triggered. This strategy will use gentle language to convey the negative impact of violent behavior, while guiding the topic towards a safer and more positive direction, thus strengthening content protection for minors. If the subject is determined to be an adult, and the interaction content also involves topics related to moderate violence, a neutral intervention strategy will be adopted. This strategy will not actively reinforce or criticize the topic, but will only control the boundaries of the dialogue through objective expression to prevent the topic from escalating to extremes or spreading, balancing the expressive needs of adult users with the bottom line of content safety. When the risk control system detects that the interaction content touches on issues related to severe violence, regardless of the user profile dimensions (age, identity, etc.), a rigid prevention mechanism will be activated to immediately interrupt the interaction on the relevant topic, simultaneously push compliance prompts, strictly block the spread of severe violent content, and ensure compliance with content safety management standards.

[0084] Understandably, after the response content passes security review and is returned, this embodiment can manage and maintain the memory storage area and adjust the user profile vector in real time. Specifically, an asynchronous task is initiated to update long-term memory, packaging the target session request and the generated response content, calling LLM to perform a dialogue summary, generating key information points (such as "the user requested a picture of a cat in this round of dialogue and expressed that they liked it"). Subsequently, the summary is converted into a vector by the vector model and stored in the user's vector database along with the summary text for future dialogue retrieval. Next, according to a preset memory management strategy, several memory fragments already existing in the memory storage area are managed. Finally, after obtaining the user's response behavior to the response content, the user profile vector is adjusted based on the response behavior, updating the relevant dimension data in the vector.

[0085] It should be noted that several memory segments in the memory storage area have corresponding metadata information; the metadata information includes storage timestamp, last access timestamp, and memory segment weight; the lifetime of several cached memory segments in the first memory storage area is positively correlated with the user subscription level; the upper limit of the number of memory segments stored in the second memory storage area is positively correlated with the user subscription level; and the upper limit of the memory retrieval frequency of the memory storage area within a preset time period is positively correlated with the user subscription level. Accordingly, the processing flow for managing the memory storage area is as follows: If there is a memory fragment to be strengthened in the enhanced memory fragment that has a positive feedback effect on the current dialogue interaction, then the weight of the memory fragment to be strengthened is increased according to a preset logarithmic function with time decay; several memory fragments in the memory storage area are scanned based on a preset memory scanning frequency, and the weight of the memory fragment to be decayed among the several memory fragments is reduced based on a preset decay factor; when a memory fragment to be eliminated is detected in the memory storage area, the memory fragment to be eliminated is eliminated; the memory fragment to be decayed is a memory fragment whose difference between the last access timestamp and the current timestamp is greater than a preset difference threshold; the memory fragment to be eliminated is a memory fragment whose weight is lower than a preset memory fragment weight threshold; wherein, the preset difference threshold is positively correlated with the user subscription level; the preset memory fragment weight threshold is negatively correlated with the user subscription level. That is, each memory fragment in the memory storage area contains metadata information including the storage timestamp of creation, the last access timestamp, and dynamic memory fragment weight. If the enhanced memory fragments contain memory fragments that have a positive feedback effect on the current dialogue interaction, that is, if the memory fragment is successfully retrieved and has a positive impact on subsequent dialogue (which can be determined through user feedback or model judgment), then the weight of the memory fragment is increased by a preset logarithmic function with time decay, making it easier to recall in future retrievals. At the same time, an asynchronous background task scans the memory storage area according to a preset memory scanning frequency. For memory fragments whose difference between the last access timestamp and the current timestamp is greater than a preset difference threshold, that is, memory fragments that have not been accessed for a long time, their weights are gradually reduced based on a preset decay factor. If a memory fragment whose weight is lower than the preset memory fragment weight threshold is detected, it is archived or deleted, thereby freeing up space for new and more important memories and maintaining the efficiency and relevance of the memory bank.

[0086] In addition, this embodiment introduces a flexible memory payment mechanism, where a user's payment level is directly mapped to the specific technical parameters of their memory system, thereby achieving differentiated resource allocation. Specifically, the user's subscription level (e.g., free, standard, advanced) is associated with a set of configuration parameters, including: vector storage quota: the maximum number of memory fragments allowed to be stored in the vector database; cache TTL (Time-to-Live): advanced users have a longer short-term memory cache time; memory decay threshold: paid users experience slower memory decay and a lower eviction threshold, thus "remembering" information for longer periods; and API call frequency: limiting the call frequency of memory modules to ensure fair allocation of system resources. For example, advanced users have unlimited memory module calls, while ordinary users are limited to 10 calls per day. That is, the lifetime of cached memory fragments in the first memory storage area, the maximum number of memory fragments stored in the second memory storage area, and the maximum memory retrieval frequency of the memory storage area within a preset time period are all positively correlated with the user's subscription level. Simultaneously, the preset difference threshold is positively correlated with the user's subscription level, and the preset memory fragment weight threshold is negatively correlated with the user's subscription level. In other words, the higher the subscription level, the longer the lifespan, the higher the storage limit, the higher the retrieval frequency limit, the slower the memory decay rate, and the lower the elimination threshold.

[0087] It should be noted that the processing flow for adjusting the user profile vector is as follows: The response behavior to the reply content is obtained, and based on the target contextualized personality instruction, the response behavior is analyzed to determine the feedback type of the response behavior according to the analysis results. The weights of relevant tags in the user profile vector are then adjusted according to the feedback type. The feedback type includes positive feedback and negative feedback. Alternatively, based on a preset vector update frequency, user interaction data within a preset historical time period is used to adjust the tag weights in the user profile vector. That is, firstly, the user's response behavior to the reply content is obtained through the target interaction interface. Then, combined with the target contextualized personality instruction, the response behavior is analyzed to determine the feedback type of the response behavior to the current interaction. For example, if the user's reply becomes longer and more positive after adopting the casual_and_encouraging style, it is considered a positive feedback signal. Conversely, if the user's reply is short and negative (such as "Don't bother me," "Get to the point"), it is a negative feedback signal. Finally, based on the determined feedback type, a fine-tuning of relevant parameters in the user profile vector is triggered. Using a logic similar to Bayesian inference, new interactions are treated as "evidence" to update the judgment of user preferences (prior beliefs), adjusting the components of relevant parameters in the user profile vector. For example, negative feedback will reduce the probability weight of choosing the casual_and_encouraging style in the future. In addition, when a user generates new key interactive behaviors (such as the first use of the "voice generation" function, or feedback of "disliking technical terms"), the module extracts data in real time and updates the corresponding tags (e.g., adding "Common Functions - Text-to-Image Generation" to "Common Functions - Text-to-Image Generation / Voice Generation", and adjusting "Interaction Sensitivity - High" to "Interaction Sensitivity - High (Dislike of Technical Terms)"). Alternatively, based on a preset vector update frequency, such as every morning, the tags of all users can be iterated. Based on the interaction data within a preset historical time period, such as the interaction data of the past 24 hours, low-weight tags (such as "interest topic - fitness" that has not been triggered in 30 days) can be downgraded or deleted, while high-frequency triggered tags (such as "workplace tips generation" called for 7 consecutive days) can be weighted up to ensure that the tag system is concise and accurate.

[0088] In this way, this embodiment evaluates emotional states through a multimodal emotion classification model, which can capture emotional details more comprehensively and accurately, providing a reliable basis for modality weight adjustment. Mapping emotional states to emotion vectors and using emotional valence components as the core basis for adjusting modality weights allows the output modality to better match the user's current emotional needs, further enhancing the emotional resonance and satisfaction of user interaction. Combining the request intent and the adjusted modality weights to call the response generation model makes the generated response more targeted, accurately matching the core user needs and conforming to the user's current emotional modality in terms of presentation, thus improving the overall interactive experience. A dual security review mechanism is adopted, combining local algorithms and external services to conduct security reviews of the response content before returning it, which ensures both the efficiency of the review and improves its comprehensiveness and authority, effectively filtering out illegal and inappropriate content. When the review fails, differentiated security prompts are provided based on user group type and risk control level, ensuring both the rigor of risk control and the consideration of user experience. The memory is updated based on the current interaction's request and response. The storage area ensures the timeliness and continuity of memory information, avoiding response deviations caused by memory lag, while the asynchronous design ensures low latency in user response. The association between storage area rules and user subscription levels enables differentiated memory services, improving the satisfaction of high-value users. Positive feedback segments are weighted using a time-decay logarithmic function, reinforcing frequently used memory information while preventing resource waste caused by unlimited weight growth. Weights are decayed for segments that haven't been accessed for a long time, and low-weight segments are discarded. Combined with threshold settings associated with subscription levels, this ensures that memory storage for low-subscription-level users is not redundant, while reserving more potentially valuable memory segments for high-subscription-level users, balancing resource consumption and memory effectiveness. Analyzing feedback types using contextualized personality instructions allows for more accurate feedback type judgment, avoiding bias. Adjusting user profile vector parameters based on feedback type enables the profile to dynamically adapt to the user's latest interaction attitudes and changing needs, improving long-term user satisfaction and stickiness.

[0089] As can be seen from the above, the multimodal intelligent anthropomorphic interactive system of this application performs memory retrieval on the target session request initiated by the target client through a preset memory retrieval mechanism, extracts the relevant target memory fragments from the memory storage area, and obtains enhanced memory fragments through preset knowledge graph enhancement. Combined with the user's basic information and user's historical memory corresponding to the target session request, a user profile vector including basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags is constructed. This vector is used to fill a preset personality instruction template to generate a target contextualized personality instruction. Based on the enhanced memory fragments, the target contextualized personality instruction, the emotional state of the target session request, and the request itself, intent recognition is performed to obtain the request intent. The weights of each output modality are adjusted according to the emotional vector mapped by the emotional state. Then, the response generation model is called to generate response content according to the request intent and the modality weights. After security review, the response is returned. In this way, through the above-described process of this application embodiment, relevant target memory fragments are extracted through a memory retrieval mechanism. A user profile vector is constructed by combining user basic information and historical memory, and contextualized personality instructions are generated. This allows conversational interaction to fully integrate past user information and personalized traits, making responses more aligned with user habits and needs. After obtaining the target memory fragments, they are enhanced using a pre-set knowledge graph to achieve semantic expansion and entity association. This not only solves the problem of information fragmentation that may exist in the original memory fragments but also endows the system with cross-conversation reasoning capabilities, avoiding users repeatedly explaining their needs and further optimizing the coherence and professionalism of long-memory conversations. Based on multi-dimensional information such as emotional state and memory fragments, the request intent is identified, significantly improving intent judgment compared to identifying only based on request content. Accuracy is improved, reducing misunderstandings caused by incomplete information; emotional states are transformed into emotional vectors and the output modality weights are adjusted to ensure responses match the user's current emotions, enhancing emotional resonance in the interaction; responses undergo security review before being returned, ensuring the security of interactive content and preventing the transmission of harmful information; memory storage areas are updated and memory fragments are managed based on conversation requests and response content, while user profile vectors are adjusted according to user response behavior, continuously accumulating user information and optimizing user profiles. As the number of interactions increases, the accuracy and personalization of responses continuously improve, further enhancing users' trust in and experience of the multimodal intelligent anthropomorphic interactive system, thereby solving the problems of short memory, single interaction modality, and mechanical response content that are common in existing conversational interaction methods.

[0090] Accordingly, see Figure 5 As shown in the embodiments of this application, a multi-terminal long-memory multimodal intelligent anthropomorphic interactive system is also provided, including:

[0091] The memory retrieval module 11 is used to perform memory retrieval on the target session request initiated by the target client using a preset memory retrieval mechanism, so as to obtain the relevant target memory fragments from the memory storage area, and to enhance the target memory fragments using a preset knowledge graph to obtain enhanced memory fragments.

[0092] The template filling module 12 is used to construct a corresponding user profile vector based on the user's basic information and user history memory corresponding to the target session request, and to fill the preset personality instruction template with the user profile vector to generate the target contextualized personality instruction; the user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags and topic preference tags;

[0093] The intent recognition module 13 is used to recognize the request intent based on the enhanced memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, so as to obtain the corresponding request intent;

[0094] The weight adjustment module 14 is used to adjust the modal weights of each output modality according to the emotion vector mapped by the emotion state, so as to call the corresponding response generation model according to the request intent and the modal weights, generate the response content of the target session request, and return the response content after it passes the security review.

[0095] In some specific embodiments, the memory storage area includes a first memory storage area and a second memory storage area; the first memory storage area is a preset memory cache area; and the second memory storage area is a preset vector database for storing memory fragments.

[0096] In some specific embodiments, the memory retrieval module 11 may specifically include:

[0097] A memory retrieval unit is configured to perform a retrieval in the first memory storage area based on the user identifier corresponding to the target session request, so as to determine candidate memory segments with the user identifier as the key;

[0098] The similarity determination unit is used to encode the target session request into a target query vector using a preset vector transformation model, and to determine the first cosine similarity between the candidate memory fragment and the target query vector;

[0099] The full-text retrieval unit is used to select a target historical memory fragment from the second memory storage area based on the second cosine similarity between the historical memory fragment and the target query vector when the first cosine similarity satisfies the preset vector retrieval conditions, and to perform keyword full-text retrieval based on the target session request when the current preset mixed retrieval conditions are met, so as to obtain the corresponding retrieval results.

[0100] The fragment determination unit is used to reorder the search results and the target historical memory fragments according to the target semantic association, and determine the relevant target memory fragments; the target semantic association is the semantic association between the target session request and the search results and the target historical memory fragments respectively.

[0101] In some specific embodiments, several memory segments in the memory storage area have corresponding metadata information; the metadata information includes a storage timestamp, a last access timestamp, and a memory segment weight; the lifetime of several cached memory segments in the first memory storage area is positively correlated with the user's subscription level; the upper limit of the number of memory segments stored in the second memory storage area is positively correlated with the user's subscription level; the upper limit of the memory retrieval frequency of the memory storage area within a preset time period is positively correlated with the user's subscription level;

[0102] Accordingly, the multi-terminal long-memory multimodal intelligent anthropomorphic interactive system may further include:

[0103] The weight enhancement unit is used to enhance the weight of the memory fragment to be enhanced according to a preset logarithmic function with time decay if there is a memory fragment to be enhanced in the enhanced memory fragment that has a positive feedback effect on the current dialogue interaction.

[0104] A segment scanning unit is used to scan a plurality of memory segments in the memory storage area based on a preset memory scanning frequency, so as to reduce the weight of the memory segment to be decayed among the plurality of memory segments based on a preset decay factor, and when a memory segment to be eliminated is detected in the memory storage area, the memory segment to be eliminated is eliminated; the memory segment to be decayed is a memory segment whose difference between the last access timestamp and the current timestamp is greater than a preset difference threshold; the memory segment to be eliminated is a memory segment whose weight is lower than a preset memory segment weight threshold;

[0105] The preset difference threshold is positively correlated with the user subscription level; the preset memory fragment weight threshold is negatively correlated with the user subscription level.

[0106] In some specific embodiments, the memory retrieval module 11 may specifically include:

[0107] A relationship matching unit is used to identify the target entity in the target session request and match the predefined entity relationship of the target entity using a preset knowledge graph;

[0108] An entity binding unit is used to determine the relevant memory fragments in the target memory fragments that are related to the target entity, and to associate and bind the relevant entities in the relevant memory fragments that correspond to the target entity with the preset knowledge graph, so as to extract the entity association information between the target entities based on the obtained binding relationship and the predefined entity relationship;

[0109] An information appending unit is used to append the entity association information as extended semantic information to the target memory fragment to obtain an enhanced memory fragment.

[0110] In some specific embodiments, the template filling module 12 may specifically include:

[0111] The data extraction unit is used to determine the user basic information corresponding to the target session request, and extract the corresponding structured data from the user basic information, so as to generate the corresponding basic attribute tags based on the structured data and the location of the target client;

[0112] The data analysis unit is used to statistically analyze user interaction behavior data from historical memory in order to generate behavioral preference tags based on the analysis results.

[0113] The topic supplementation unit is used to perform topic clustering on the dialogue content in the user's historical memory using a preset topic modeling algorithm, and to supplement topics based on the user's historical search information to obtain corresponding topic preference tags.

[0114] The sentiment analysis unit is used to perform sentiment analysis on the user's historical memories and corresponding historical feedback behaviors using a preset sentiment classification model, so as to generate corresponding sentiment feature labels.

[0115] A vector construction unit is used to construct a corresponding user profile vector based on the basic attribute labels, the behavioral preference labels, the topic preference labels, and the emotional feature labels.

[0116] The template filling unit is used to fill the preset personality instruction template with the user profile vector and the emotional state through a preset dynamic content filling engine, so as to generate the target contextualized personality instruction.

[0117] In some specific embodiments, the weight adjustment module 14 may specifically include:

[0118] The state mapping unit is used to evaluate the emotional state using a preset multimodal emotion classification model, and to map the emotional state into an emotion vector based on the evaluation results.

[0119] The weighting unit is used to increase the modal weight of the non-text output modality if the emotional valence component in the emotion vector is lower than a preset emotional valence component threshold; the non-text output modality includes image output modality and audio output modality;

[0120] The weight control unit is configured to control the modal weight of the text output modality to be higher than the modal weight of the non-text output modality if the emotional valence component in the emotional vector is not lower than the preset emotional valence component threshold.

[0121] In some specific embodiments, the weight adjustment module 14 may specifically include:

[0122] The content review unit is used to perform security review operations on the response content by using a local preset sensitive word recognition algorithm and a content review service called through a preset security review interface;

[0123] The level determination unit is used to determine the corresponding user group type based on the user profile vector and determine the risk control level of the current interaction if the security audit operation fails, so as to provide corresponding security prompt feedback for the target session request based on the risk control level and the user group type.

[0124] The content return unit is used to format the response content and return the processed response content if the security audit operation has passed.

[0125] In some specific embodiments, the multi-terminal long-memory multimodal intelligent anthropomorphic interactive system may further include:

[0126] The behavior analysis unit is used to acquire response behaviors to the response content, and analyze the response behaviors based on the target contextualized personality instructions, so as to determine the feedback type of the response behavior according to the analysis results, and adjust the weight of relevant tags in the user profile vector according to the feedback type; the feedback type includes positive feedback type and negative feedback type;

[0127] Alternatively, a weight adjustment unit is used to adjust the label weights in the user profile vector based on a preset vector update frequency and using user interaction data within a preset historical time period.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0129] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or article that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or article. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or article that includes said element.

[0130] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-terminal, long-memory, multimodal intelligent anthropomorphic interaction method, characterized in that, Applications include multimodal intelligent anthropomorphic interactive systems, including: A preset memory retrieval mechanism is used to retrieve the target session request initiated by the target client to obtain the relevant target memory fragment from the memory storage area. The target memory fragment is then enhanced using a preset knowledge graph to obtain the enhanced memory fragment. Based on the user's basic information and historical memory corresponding to the target session request, a corresponding user profile vector is constructed, and the user profile vector is used to fill the preset personality instruction template to generate the target contextualized personality instruction; the user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags; Based on the enhanced post-memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, the request intent is identified to obtain the corresponding request intent; The modality weights of each output modality are adjusted according to the emotion vector of the emotion state mapping, so as to call the corresponding response generation model according to the request intent and the modality weights, generate the response content of the target session request, and return it after the response content passes the security review.

2. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 1, characterized in that, The memory storage area includes a first memory storage area and a second memory storage area; the first memory storage area is a preset memory cache area; the second memory storage area is a preset vector database for storing memory fragments.

3. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 2, characterized in that, The step of using a preset memory retrieval mechanism to perform memory retrieval on a target session request initiated by a target client, in order to obtain relevant target memory fragments from the memory storage area, includes: Based on the user identifier corresponding to the target session request, a search is performed in the first memory storage area to determine candidate memory segments with the user identifier as the key; The target session request is encoded into a target query vector using a preset vector transformation model, and the first cosine similarity between the candidate memory fragment and the target query vector is determined. When the first cosine similarity satisfies the preset vector retrieval conditions, the target historical memory fragment is selected from the second memory storage area based on the second cosine similarity between the historical memory fragment and the target query vector. When the preset mixed retrieval conditions are met, keyword full-text retrieval is performed based on the target session request to obtain the corresponding retrieval results. The search results and the target historical memory fragments are reordered based on the target semantic association to determine the relevant target memory fragments; the target semantic association refers to the semantic association between the target session request and the search results and the target historical memory fragments, respectively.

4. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 3, characterized in that, The memory segments in the memory storage area have corresponding metadata information; the metadata information includes storage timestamp, last access timestamp, and memory segment weight; the lifetime of several cached memory segments in the first memory storage area is positively correlated with the user subscription level; the upper limit of the number of memory segments stored in the second memory storage area is positively correlated with the user subscription level; The upper limit of the memory retrieval frequency in the memory storage area within a preset time period is positively correlated with the user's subscription level; Accordingly, after the response content passes the security audit and is returned, the process also includes: If there is a memory fragment to be strengthened in the enhanced memory fragment that has a positive feedback effect on the current dialogue interaction, then the weight of the memory fragment to be strengthened is increased according to a preset logarithmic function with time decay. A number of memory segments in the memory storage area are scanned based on a preset memory scanning frequency. The weight of the memory segment to be decayed among the memory segments is reduced based on a preset decay factor. When a memory segment to be eliminated is detected in the memory storage area, the memory segment to be eliminated is eliminated. The memory segment to be decayed is the memory segment whose difference between the last access timestamp and the current timestamp is greater than a preset difference threshold. The memory segment to be eliminated is the memory segment whose weight is lower than a preset memory segment weight threshold. The preset difference threshold is positively correlated with the user subscription level; the preset memory fragment weight threshold is negatively correlated with the user subscription level.

5. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 1, characterized in that, The process of enhancing the target memory fragment using a preset knowledge graph to obtain the enhanced memory fragment includes: Identify the target entity in the target session request, and use a preset knowledge graph to match the predefined entity relationships of the target entity; Identify the relevant memory fragments in the target memory fragments that are related to the target entity, and associate and bind the relevant entities in the relevant memory fragments that correspond to the target entity with the preset knowledge graph, so as to extract the entity association information between the target entities based on the obtained binding relationship and the predefined entity relationship; The entity association information is appended to the target memory fragment as extended semantic information to obtain the enhanced memory fragment.

6. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 1, characterized in that, The step of constructing a corresponding user profile vector based on the user's basic information and historical memory corresponding to the target session request, and using the user profile vector to fill in a preset personality instruction template to generate a target contextualized personality instruction, includes: Determine the user basic information corresponding to the target session request, and extract the corresponding structured data from the user basic information to generate corresponding basic attribute tags based on the structured data and the location of the target client; Statistical analysis of user interaction behavior data in historical memory is used to generate behavioral preference labels based on the analysis results; A preset topic modeling algorithm is used to perform topic clustering on the dialogue content in the user's historical memory, and topics are supplemented based on the user's historical search information to obtain corresponding topic preference tags; A preset sentiment classification model is used to perform sentiment analysis on the user's historical memories and corresponding historical feedback behaviors to generate corresponding sentiment feature labels. Construct a corresponding user profile vector based on the basic attribute tags, the behavioral preference tags, the topic preference tags, and the emotional feature tags; By using a preset dynamic content filling engine, the preset personality instruction template is filled with the user profile vector and the emotional state to generate target contextualized personality instructions.

7. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 1, characterized in that, The step of adjusting the modal weights of each output modality based on the emotion vector mapped from the emotion state includes: The emotional state is evaluated using a pre-defined multimodal emotion classification model, and the emotional state is mapped into an emotion vector based on the evaluation results. If the emotional valence component in the emotional vector is lower than a preset emotional valence component threshold, then the modal weight of the non-text output modality is increased; the non-text output modality includes the image output modality and the audio output modality. If the emotional valence component in the emotional vector is not lower than the preset emotional valence component threshold, then the modal weight of the text output modality is controlled to be higher than the modal weight of the non-text output modality.

8. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to claim 1, characterized in that, The process of returning the response after the content has passed security review includes: The response content is subjected to security review using a local preset sensitive word recognition algorithm and a content review service called through a preset security review interface; If the security audit operation fails, the corresponding user group type is determined based on the user profile vector, and the risk control level of the current interaction is determined. Based on the risk control level and the user group type, corresponding security prompts and feedback are given to the target session request. If the security audit has been approved, the response content will be formatted and returned.

9. The multi-terminal long-memory multimodal intelligent anthropomorphic interaction method according to any one of claims 1 to 8, characterized in that, After the response content passes security review and is returned, the process also includes: The system acquires response behaviors to the given answers and analyzes these behaviors based on the target contextualized personality instructions. It then determines the feedback type of the response behavior based on the analysis results and adjusts the weights of relevant tags in the user profile vector according to the feedback type. The feedback types include positive feedback and negative feedback. Alternatively, based on a preset vector update frequency, the label weights in the user profile vector can be adjusted using user interaction data within a preset historical time period.

10. A multi-terminal, long-memory, multimodal intelligent anthropomorphic interactive system, characterized in that, include: The memory retrieval module is used to perform memory retrieval on the target session request initiated by the target client using a preset memory retrieval mechanism, so as to obtain the relevant target memory fragments from the memory storage area, and to enhance the target memory fragments using a preset knowledge graph to obtain enhanced memory fragments. The template filling module is used to construct a corresponding user profile vector based on the user's basic information and user history memory corresponding to the target session request, and to fill the preset personality instruction template with the user profile vector to generate the target contextualized personality instruction. The user profile vector includes basic attribute tags, behavioral preference tags, emotional feature tags, and topic preference tags; The intent recognition module is used to identify the request intent based on the enhanced post-memory fragment, the target contextualized personality instruction, the emotional state of the target conversation request, and the target conversation request, so as to obtain the corresponding request intent; The weight adjustment module is used to adjust the modal weights of each output modality according to the emotion vector mapped by the emotion state, so as to call the corresponding response generation model according to the request intent and the modal weights, generate the response content of the target session request, and return the response content after it passes the security review.

Images