Intelligent interaction method and device and electronic equipment

Abstract

The invention provides an intelligent interaction method and apparatus, and an electronic device. The method comprises the steps of receiving current session data sent by a terminal device; on the basis of a personality model created for a preset user, personalized answer content for the current session data is generated, the personality model is constructed on the basis of multi-modal data of the preset user, and the preset user is an object simulated by a corresponding user intention of the terminal device; and sending the personalized answer content to the terminal device. In the implementation process of the scheme, personalized answering is performed on the current session data sent by the terminal equipment according to the current session data through the personality model constructed based on the multi-modal data, so that the cross-modal personalized service capability is realized, the interaction system can generate highly anthropomorphic response, and the user experience is improved. The limitation that a traditional dialogue system is only based on rule or simple semantic matching is broken through, and therefore continuity and consistency of sessions are effectively kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, digital twinning and personality modeling, in particular to an intelligent interaction method and device and electronic equipment. BACKGROUND

[0002] The current intelligent dialogue system generally adopts a single mode or static interaction architecture, mainly relying on text data processing and preset rule dialogue logic. However, in the multi-modal scene of medical consultation or emotional support, the system may generate irrelevant or contradictory replies due to the neglect of user historical medical history or emotional changes. Therefore, the current intelligent dialogue system is difficult to maintain the coherence and consistency of the conversation. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide an intelligent interaction method, device and electronic equipment, which can improve the problem that the current intelligent dialogue system is difficult to maintain the coherence and consistency of the conversation.

[0004] The embodiments of the present application provide an intelligent interaction method, comprising: receiving current conversation data sent by a terminal device; generating personalized answer content for the current conversation data based on a personality model created for a preset user, the personality model being constructed based on multi-modal data of the preset user, the preset user being an object simulated by a corresponding user of the terminal device; and sending the personalized answer content to the terminal device. In the implementation process of the above scheme, the current conversation data sent by the terminal device is answered personally based on the personality model constructed based on the multi-modal data according to the current conversation data, realizing the personalized service ability across the modal, making the interaction system be able to produce highly personalized response, breaking through the limitation of the traditional dialogue system based on rules or simple semantic matching, thereby effectively maintaining the coherence and consistency of the conversation.

[0005] Optionally, in the embodiments of the present application, before generating the personalized answer content for the current session data based on the personality model created for the preset user, the method further comprises: receiving multi-modal data of the preset user; extracting structured information from the multi-modal data; constructing the personality model based on the structured information; and fine-tuning and prompting the base large model to configure the base large model to be able to call the personality model to generate the personalized answer content. In the implementation process of the above scheme, the base large model is fine-tuned and prompted to configure the base large model, which realizes flexible calling and optimization of the base large model, so that the base large model can efficiently generate personalized answers in combination with the personality model. This not only improves the response speed of the model, but also significantly reduces the consumption of computing resources. Further, the combination of the personality model and the base large model enables the generated answer content to not only have a high degree of personalization, but also maintain the extensive knowledge coverage and language generation capability of the base large model, thereby achieving a good balance between personalization and universality. In addition, through the comprehensive utilization of multi-modal data and the extraction of structured information, the technology can more deeply understand the user's context and intent, thereby generating more context-related and coherent personalized answers, improving user experience and interaction quality.

[0006] Optionally, in the embodiments of the present application, the personality model includes a personal knowledge base; and the generating of the personalized answer content for the current session data comprises: extracting associated information from the current session data and historical session data of the terminal device; identifying a session intent of the current session data according to the personality model and the associated information; retrieving relevant information from the personal knowledge base based on the session intent; and generating the personalized answer content according to the retrieval result. In the implementation process of the above scheme, by combining the current session data with the historical session data to extract associated information, the context and requirements of the user can be more comprehensively understood, thereby significantly improving the accuracy and relevance of the personalized answer, because the system not only considers the current input, but also utilizes the implicit information in the historical interaction. Further, by utilizing the personality model and the associated information to identify the session intent, the system can more accurately capture the real intent of the user, avoiding the intent misjudgment caused by the lack of context or personalized information in traditional methods, and retrieving relevant information from the personal knowledge base based on the session intent ensures the high personalization and pertinence of the answer content, because the personal knowledge base stores data and preferences directly related to the user, which can provide answers more in line with the user's needs. In addition, generating the personalized answer content according to the retrieval result enables the system to dynamically adjust the answering strategy, not only meeting the user's immediate needs, but also optimizing the answering method according to the historical interaction mode, thereby improving user experience and satisfaction.

[0007] Optionally, in the embodiments of the present application, the personalized answer content for the current session data is generated, comprising: if the current session data contains multi-modal content, embedding encoding the multi-modal content by a multi-modal encoder in the AI dialogue engine to obtain an embedding representation vector, the multi-modal encoder comprising a text encoder, an image encoder, an audio encoder and / or a video encoder; generating the personalized answer content for the current session data according to the embedding representation vector by the personality model. In the implementation process of the above scheme, the multi-modal encoder uniformly embeds and encodes various modal contents such as text, image, audio and video, which can convert different modal session data into unified vector representation, thereby realizing the fusion processing of multi-modal data, solving the limitation of traditional dialogue system that can only process single modal data, and significantly improving the understanding ability of the system to complex conversation scenarios. Further, by using the personality model to generate personalized answer content according to the embedding representation vector, the system can dynamically adjust the answering mode according to the conversation style and preference of the user, thereby realizing a more natural and personalized interactive experience, overcoming the shortcomings of traditional dialogue system that the answer content is single and lacks personalization. In addition, through the collaborative work of the multi-modal encoder and the personality model, the system can simultaneously consider the multi-modal characteristics and personalized needs of the session data, thereby generating more accurate and appropriate answers in complex conversation scenarios, improving the overall performance and user satisfaction of the dialogue system.

[0008] Optionally, in the embodiments of the present application, the personality model further comprises a character emotion model, a language style model and / or an appearance sound model; after generating the personalized answer content for the current session data based on the personality model created for the preset user, the method further comprises: analyzing the emotional state of the current session data through the character emotion model, and adjusting the emotional tendency of the personalized answer content according to the emotional state; and / or, analyzing the language style of the current session data through the language style model, and controlling the word usage habit, speech speed feature or dialect proportion of the personalized answer content according to the language style; and / or, generating a virtual appearance image and voice sound data corresponding to the personalized answer content through the appearance sound model. In the implementation process of the above scheme, by introducing the character emotion model to analyze the emotional state of the current session data and adjust the emotional tendency of the personalized answer content, the real-time perception and response of the dialogue system to the user's emotion can be realized, and by introducing the language style model to analyze the language style of the current session data and control the word usage habit, speech speed feature or dialect proportion of the personalized answer content, the high consistency of the dialogue system and the user in the language expression level can be realized. Further, by introducing the appearance sound model to generate a virtual appearance image and voice sound data corresponding to the personalized answer content, the multi-modal output capability of the dialogue system can be realized, and the text answer can be expanded to a comprehensive interactive experience containing visual and auditory elements. This technical effect is derived from the ability to map the features of the personality model to multi-modal forms, so that the system can further strengthen the personalized expression through virtual images and voice features, and improve the immersion and affinity of the interaction.

[0009] Optionally, in the embodiments of the present application, after sending the personalized answer content to the terminal device, the method further comprises: receiving user feedback data sent by the terminal device, the user feedback data being fed back by the corresponding user of the terminal device according to the personalized answer content; updating the personal knowledge base according to the user feedback data to obtain an updated personal knowledge base; and incrementally training the personality model based on the updated personal knowledge base.

[0010] In the implementation process of the above scheme, the dynamic optimization of the knowledge base is realized by receiving user feedback data and updating the personal knowledge base based thereon, so that the system can continuously adapt to the changes in user needs, thereby improving the real-time and accuracy of the answer content. This closed-loop feedback mechanism ensures the self-evolution ability of the knowledge base, avoiding the lag or outdated problems caused by static knowledge base. Further, based on the incremental training of the personality model based on the updated personal knowledge base, the personality model can continuously learn and adapt to the preferences and behavior patterns of the user, thereby improving the personalized recommendation ability of the model. This continuous training mechanism avoids the degradation of the model performance, ensuring the long-term effectiveness of the model in practical applications. In addition, combined with the update of the user feedback data and the personal knowledge base, the personality model is optimized in a targeted manner, so that the model can more accurately capture the personalized needs of the user, thereby improving the user satisfaction and interaction experience. This fine-grained optimization strategy avoids the generalization deficiency or overfitting problem in traditional model training.

[0011] Optionally, in the embodiments of the present application, the corresponding user of the preset user and the terminal device is a family relationship, an identity continuation relationship, an emotional support relationship or an idealized relationship. In the implementation process of the above scheme, by limiting the corresponding user of the preset user and the terminal device to a family relationship, an identity continuation relationship, an emotional support relationship or an idealized relationship, the system can more accurately identify and predict the emotional needs and interaction patterns of the user, because these specific relationships often have more stable and predictable behavior characteristics, thereby improving the accuracy and individualization level of the interaction.

[0012] The embodiments of the present application also provide an intelligent interaction device, comprising: a conversation data receiving module for receiving current conversation data sent by a terminal device; an answer content generating module for generating personalized answer content for the current conversation data based on a personality model created for a preset user, the personality model being constructed based on multi-modal data of the preset user, the preset user being an object simulated by a corresponding user of the terminal device; and an answer content sending module for sending the personalized answer content to the terminal device.

[0013] Optionally, in the embodiments of the present application, the intelligent interaction device further comprises: a modal data receiving module for receiving multi-modal data of the preset user; a structured information extracting module for extracting structured information from the multi-modal data; a personality model constructing module for constructing the personality model based on the structured information; and a personality model calling module for fine-tuning and prompting an engineering unit configuration base large model to enable it to call the personality model to generate personalized answer content.

[0014] Optionally, in the embodiment of the present application, the personality model comprises a personal knowledge base; the answer content generation module comprises: an associated information extraction submodule, configured to extract associated information from the current session data and historical session data of the terminal device; a session intention recognition submodule, configured to recognize a session intention of the current session data according to the personality model and the associated information; a related information retrieval submodule, configured to retrieve related information from the personal knowledge base based on the session intention; and an answer content generation submodule, configured to generate personalized answer content according to the retrieval result.

[0015] Optionally, in the embodiment of the present application, the answer content generation module comprises: an embedding representation obtaining submodule, configured to, if the current session data comprises multi-modal content, perform embedding coding on the multi-modal content by a multi-modal encoder in the AI dialogue engine to obtain an embedding representation vector, the multi-modal encoder comprising a text encoder, an image encoder, an audio encoder and / or a video encoder; and an answer content generation submodule, configured to generate personalized answer content for the current session data according to the embedding representation vector by the personality model.

[0016] Optionally, in the embodiment of the present application, the personality model further comprises: a personality emotion model, a language style model and / or an appearance and voice model; the intelligent interaction device further comprises: a state tendency adjustment module, configured to analyze an emotional state of the current session data by the personality emotion model, and adjust an emotional tendency of the personalized answer content according to the emotional state; and / or a language style control module, configured to analyze a language style of the current session data by the language style model, and control a word usage habit, a speech speed feature or a dialect proportion of the personalized answer content according to the language style; and / or an image data generation module, configured to generate a virtual appearance image and voice data corresponding to the personalized answer content by the appearance and voice model.

[0017] Optionally, in the embodiment of the present application, the intelligent interaction device further comprises: a feedback data receiving module, configured to receive user feedback data sent by the terminal device, the user feedback data being fed back by a corresponding user of the terminal device according to the personalized answer content; a personal knowledge base updating module, configured to update the personal knowledge base according to the user feedback data to obtain an updated personal knowledge base; and a personality model training module, configured to perform incremental training on the personality model based on the updated personal knowledge base.

[0018] Optionally, in the embodiment of the present application, the corresponding user of the terminal device and the preset user are in a kinship relationship, an identity continuation relationship, an emotional support relationship or an idealized relationship.

[0019] The embodiment of the present application also provides an electronic device, comprising: a processor and a memory, the memory storing machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the above described method.

[0020] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is run by a processor to execute the method described above.

[0021] The embodiment of the present application further provides a computer program product, comprising: a computer program or computer instruction, which is run by a processor to execute the method described above. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0023] Figure 1 The flowchart of the intelligent interaction method provided by the embodiment of the present application is shown; Figure 2 The structure diagram of the intelligent interaction device provided by the embodiment of the present application is shown; Figure 3 The structure diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that the drawings in the embodiments of the present application only play the purpose of description and illustration, and do not limit the scope of protection of the embodiments of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportion. The flowchart used in the embodiments of the present application shows the operations realized according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be realized in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or removed from the flowchart under the guidance of the content of the embodiments of the present application.

[0025] In addition, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed embodiments of the present application, but only represents selected embodiments of the present application.

[0026] It can be understood that "first", "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different. In the description of the embodiments of the present application, the term "and / or" is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A existing alone, A and B existing together, and B existing alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0027] It should be noted that the intelligent interaction method provided by the embodiments of the present application can be executed by an electronic device. The electronic device herein refers to a device terminal or a server having the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant, or a mobile Internet device. The server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, which includes a mainframe, a minicomputer, and a UNIX server.

[0028] Current intelligent dialogue systems are mostly based on single-modal or static interaction architecture, whose core processing object is only text input, and which relies on preset dialogue rules or general large language models to generate responses. Such systems usually do not have a long-term memory mechanism, and each interaction is treated as an independent event, lacking the ability to continuously track user identity, historical behavior, and context evolution. At the same time, its "static" means that personality, knowledge, and style are fixed and cannot be dynamically adjusted according to user feedback or new information, resulting in a lack of personalization and adaptability in the interaction process.

[0029] In high-sensitivity scenarios such as medical consultation or emotional support, user input often contains multi-modal information (such as emotional fluctuations in speech, uploaded examination report images, and text description of symptom development), and is highly dependent on historical context, such as past medical history, medication records, and emotional state changes. However, existing systems, due to only processing the text content of the current turn, are difficult to effectively associate key information provided by the user early on. When the user mentions "the medicine I took last time made me dizzy", and the system does not remember which visit or what kind of medicine "last time" refers to, it may recommend the same medicine or give contradictory advice; in emotional dialogue, if the system ignores the user's expression of depression tendency in several previous turns and still responds with "everything will be fine" in a light tone, it will appear cold and even cause harm.

[0030] Due to the limitation of the existing intelligent dialogue system in architecture, the intelligent dialogue system is limited to single-modal input, lacks dynamic memory and personality modeling capability, and cannot effectively integrate historical information and context evolution in multi-round and multi-modal interaction, especially in scenarios requiring deep empathy and accurate knowledge calling. Therefore, it is difficult to maintain true conversation continuity and consistency.

[0031] Please refer to Figure 1 The embodiment of the application shown provides a flowchart of an intelligent interaction method. The intelligent interaction method can be executed by an electronic device, and the specific implementation of the above intelligent interaction method can include: Step S110: receiving current session data sent by a terminal device.

[0032] The electronic device described above can receive the current session data sent by the terminal device through a graphical user interface (Graphical User Interface, GUI), a voice interface, virtual reality (VR), or augmented reality (AR) interface. Among them, the graphical user interface described above can be a chat window in a webpage or a mobile phone App, which is used to obtain image data in multi-modal data, and the voice interface is a voice input / output interface, which is used to obtain voice dialogue data in multi-modal data, Step S120: generating personalized answer content for the current session data based on a personality model created for a preset user, the personality model being constructed based on multi-modal data of the preset user, and the preset user being an object simulated by a corresponding user of the terminal device.

[0033] It can be understood that since the personality model described above is constructed and trained based on multi-modal data of the preset user, such a personality model can be regarded as a digital twin model with the personalized personality of the preset user, so as to realize natural, smooth, deep and emotionally interactive dialogue between the corresponding user of the terminal device and the digital twin model. The multi-modal data of the preset user described above can be uploaded by the corresponding user of the terminal device, which is used to train and construct a personality model that simulates the personality of a real person.

[0034] Step S130: sending the personalized answer content to the terminal device.

[0035] Similarly, the electronic device described above can also send personalized answer content to the terminal device through a graphical user interface (GUI), a voice interface, virtual reality (VR), or augmented reality (AR) interface, etc. In the above scheme, the technical means of constructing a personality model based on multi-modal data breaks through the limitations of traditional single modal text, and can capture and reproduce the characteristics of the preset user in multiple dimensions (such as voice tone, expression habit, etc.), thereby producing a more three-dimensional and realistic simulation effect in the interaction process. Further, by taking the preset user as the corresponding user intent simulation object of the terminal device, this method realizes the deep binding between the dialogue system and the specific individual, so that the system can continuously learn and adapt to the unique interaction characteristics of the user, forming a personalized interaction ability that is constantly enhanced over time.

[0036] In the implementation process of the above scheme, through the personality model constructed based on multi-modal data, the current session data sent by the terminal device is answered individually according to the current session data, so that the interaction between the corresponding user of the terminal device and the preset user has depth and temperature, realizing the cross-modal personalized service capability, so that the interaction system can produce highly personalized responses. This personality model constructed based on multi-modal data can meet the deep psychological needs of users such as emotional companionship and commemoration. Therefore, it breaks through the limitations of traditional dialogue systems based on rules or simple semantic matching, thereby effectively maintaining the coherence and consistency of the conversation.

[0037] As an optional implementation of the above intelligent interaction method, before generating the personalized answer content for the current session data based on the personality model created for the preset user, the above can also include: Step S101: receiving multi-modal data of a preset user.

[0038] The multi-modal data format described above can include: text (such as diary, email, weekly report, reading notes, chat records) of the preset user, audio (such as recorded data of birthday greetings, audio data of social media speeches), video (such as video data of family life, annual summary video data), image (such as old photos) and other multi-format data.

[0039] Step S102: extracting structured information from the multi-modal data.

[0040] It can be understood that the received multi-modal data of the preset user can be disorganized, and therefore, after receiving the multi-modal data of the preset user, a preprocessing operation can be performed on the multi-modal data, such as data cleaning, denoising, and format standardization. By receiving the multi-modal data and extracting the structured information, the technical solution can more comprehensively capture the personalized characteristics and behavior patterns of the user, thereby significantly improving the construction accuracy and integrity of the personality model, and making the generated personalized answer content more close to the real needs and preferences of the user.

[0041] After the preprocessing operation on the multi-modal data, the multi-modal data can also be data-labeled, such as using natural language processing (NLP) technology to automatically text-label (such as identifying sentiment orientation, topic classification, and identifying key entities) the text data in the multi-modal data, or using computer vision technology (CV) to automatically label, face recognition, and scene analysis the image data in the multi-modal data, or using optical character recognition (OCR) technology to perform OCR recognition and sentiment analysis on the image data, or transcribing and extracting voiceprint features from the audio data.

[0042] Step S103: constructing a personality model based on the structured information.

[0043] It can be understood that after extracting the structured information from the multi-modal data, the structured information can be used to train the personality model. The technical feature of constructing a personality model based on structured information enables the model to dynamically reflect the personalized characteristics of the user, avoiding the limitations of traditional static models, thereby realizing real-time adaptation and adjustment of user behavior and improving the timeliness and accuracy of the answer content.

[0044] Step S104: configuring the base large model through fine-tuning and prompting engineering units, so that it can call the personality model to generate personalized answer content.

[0045] The base large model described above can adopt a large language model (LLM), such as DeepSeek or GPT model, etc. In the implementation process of the above scheme, the base large model is configured through fine-tuning and prompting engineering units, which realizes flexible calling and optimization of the large model, so that it can efficiently generate personalized answers combined with the personality model, which not only improves the response speed of the model, but also significantly reduces the consumption of computing resources.

[0046] It can be understood that, since the base large model is obtained by fine-tuning the multi-modal data of the preset user and can be set with the prompt engineering unit to set the dynamic prompt engineering text of the base large model, the base large model can provide general language understanding, text understanding and text generation capabilities by calling the personality model, so that the personalized answer content generated by the personality model can conform to the knowledge, personality and style of the preset user. Further, after the base large model is configured by fine-tuning and the prompt engineering unit, the base large model can call the personality model to generate personalized answer content.

[0047] As an optional implementation of the above step S120, the personality model can include a personal knowledge base; and the implementation of generating personalized answer content for the current session data can include: Step S121: extracting associated information from the current session data and the historical session data of the terminal device.

[0048] The historical session data of the terminal device can be accumulated after recording each interaction data between the user corresponding to the terminal device and the preset user. Specifically, when the user initiates a new interaction through the terminal device (such as a mobile phone App, a webpage or a VR terminal), the electronic device first receives its current session data, which can include a text message (such as “Grandpa, do you remember when I had a fever when I was a child?”), an uploaded image (such as an old photo), a voice input (such as a confession recording) or a video clip. At the same time, the electronic device retrieves all historical accumulated interaction records between the user corresponding to the terminal device and the preset user (such as “deceased grandfather”) from the local dialogue database, including past question and answer pairs, user feedback (such as likes / dislikes), generated answers and their timestamps.

[0049] Step S122: identifying the session intent of the current session data according to the personality model and the associated information.

[0050] It can be understood that the context-related information generated in step S121 and the personality model of the preset user (including its personality tendency, language habit and typical response mode) are jointly used as context constraints and input into the multi-modal intent recognition module; the module first performs semantic encoding on the text content of the current session, and combines the embedding features of the image or audio to perform multi-label classification using the fine-tuned BERT or CLIP model, and the candidate intent includes “requesting to recall”, “seeking comfort”, “confirming facts” and the like; then, the electronic device can adjust the intent confidence according to the personality model, for example, if the preset user is a kind and gentle elder, then “do you remember …” is more inclined to be determined as emotional recall rather than fact checking, and finally outputs a high-confidence session intent label, such as “intent: request to recall (92%)”.

[0051] In the implementation of the above scheme, by combining the current session data with the historical session data to extract the associated information, the context and demand of the user can be more comprehensively understood, thereby significantly improving the accuracy and relevance of the personalized answer, because the system not only considers the current input, but also utilizes the implicit information in the historical interaction.

[0052] Step S123: retrieving relevant information from the personal knowledge base based on the session intent.

[0053] The personal knowledge base described above can store the personal facts, personal experiences, family member information, personal preferences and habits, dialect vocabulary, personal opinions and professional skills of the preset user. By utilizing the personality model and the associated information to identify the session intent, the system can more accurately capture the real intent of the user, avoid the misjudgment of the intent caused by the lack of context or personalized information in the traditional method, and retrieve relevant information from the personal knowledge base based on the session intent, ensuring the high personalization and pertinence of the answer content, because the personal knowledge base stores the data and preferences directly related to the user, which can provide answers more in line with the user's needs.

[0054] It can be understood that the session intent determined according to step S122 performs directional retrieval in the personal knowledge base of the preset user; the knowledge base is a structured database constructed by cleaning, labeling and vectorizing the multi-modal data such as diaries, recordings and photos of the preset user in the modeling stage, and each record contains an event summary, a time, an emotion label and an embedding vector; the retrieval adopts a hybrid strategy: first, candidate events are filtered through keyword matching, then the cosine similarity of the current session embedding and the knowledge item vector is calculated for sorting, and the results are filtered according to the intent type (such as "seeking comfort" returns events containing positive outcomes first); finally, one or more high-relevance knowledge segments are returned as the basis for generating personalized answer facts.

[0055] Step S124: generating personalized answer content according to the retrieval result.

[0056] It can be understood that the above retrieval result can include retrieving relevant information from the personal knowledge base, so the retrieval of relevant information from the personal knowledge base can be taken as part of the content of the context management unit, and the historical dialogue content of the user corresponding to the terminal device can be taken as another part of the content of the context management unit, and the preset prompt word engineering text is forwarded to the base large model together. The base large model calls the personality model to process the entire content of the context management unit and the preset prompt word engineering text, and returns the personalized answer content with memory continuity to the base large model. After receiving the personalized answer content with memory continuity returned by the base large model, the personality model can also send such personalized answer content with memory continuity to the terminal device.

[0057] As an optional implementation of the above step S120, the implementation of generating personalized answer content for current session data can include: Step S125: If the current session data contains multi-modal content, the multi-modal content is embedded and encoded by a multi-modal encoder in the AI dialogue engine to obtain an embedded representation vector, and the multi-modal encoder includes a text encoder, an image encoder, an audio encoder and / or a video encoder.

[0058] When the current session data sent by the user through the terminal device contains one or more modalities such as text, image, voice or video, first, the modal type of the multi-modal content is recognized by the AI dialogue engine, and the corresponding special encoder is called for processing: the text content is converted into semantic embedding through the text encoder based on Transformer (such as fine-tuned BERT or LLM encoding layer); the uploaded image extracts high-level visual features through the pre-trained visual encoder (such as ViT or ResNet); the voice input is first converted into text through automatic speech recognition (ASR), and then combined with acoustic features to generate audio embedding fused with semantics and emotion by the audio encoder (such as Wav2Vec2.0); the video is captured through the spatiotemporal modeling network (such as VideoMAE) to capture the picture content and action time sequence information at the same time. Subsequently, the embedded vectors of each modality can be integrated into a unified multi-modal embedded representation vector through a cross-modal alignment mechanism (such as multi-head attention fusion or projection to a unified semantic space), which is used for subsequent intent understanding and knowledge retrieval.

[0059] In the implementation process of the above scheme, the unified embedding coding of text, image, audio and video and other multi-modal content by the multi-modal encoder can convert the conversation data of different modalities into a unified vector representation, thereby realizing the fusion processing of multi-modal data, solving the limitation of traditional dialogue systems that can only process single modal data, and significantly improving the understanding ability of the system for complex conversation scenarios. Further, by using the personality model to generate personalized answer content according to the embedding representation vector, the system can dynamically adjust the answering manner according to the conversation style and preference of the user, thereby realizing a more natural and personalized interactive experience, overcoming the shortcomings of traditional dialogue systems that have single answer content and lack personalization.

[0060] Step S126: generating personalized answer content for the current conversation data by the personality model according to the embedding representation vector.

[0061] It can be understood that the multi-modal embedding representation vector generated in step S125 is input into the constructed personality model, which can include sub-modules such as a personal knowledge base, a personality emotion model, a language style model, etc. The embedding vector drives the knowledge base retrieval module to recall structured facts highly related to the current context; then, the personality emotion model analyzes the emotional tone of the current interaction (such as nostalgia, anxiety), and adjusts the emotional tendency of the response (such as warmth, comfort); the language style model rewrites the answer draft according to the user's language habits (such as dialect vocabulary, speech rhythm, sentence preference); finally, the above elements are injected into the base large language model (through prompt engineering or light fine-tuning), to generate personalized answer content that conforms to the facts and reflects unique personality characteristics.

[0062] As an optional implementation of the above step S120, the personality model further includes a personality emotion model, a language style model, and / or an appearance and voice model; after generating personalized answer content for the current conversation data based on the personality model created for the preset user, personalized rendering can be performed, including the following rendering methods: Method 1 Step S127: analyzing the emotional state of the current conversation data by the personality emotion model, and adjusting the emotional tendency of the personalized answer content according to the emotional state.

[0063] In the process of generating personalized answer content, the personality emotion model is used to inject emotional color that conforms to the personality characteristics of the preset user into the answer. Based on the understanding of the current conversation context, the model combines its own stable personality vector (such as mild, optimistic) to determine the overall emotional tone of the answer (such as comfort, encouragement, humor), and can be fine-tuned according to the emotional continuity in the conversation history to ensure the emotional consistency and rationality of the digital twin response.

[0064] Method 2 Step S128: Analyze the language style of the current conversation data through the language style model, and control the word usage habit, speech speed feature or dialect proportion of the personalized answer content according to the language style.

[0065] At the text level of generating the answer, the language style model is responsible for rewriting or adjusting the initial answer text generated by the base large model to conform to the preset user-specific language habits. This includes adjusting the word preference (such as using specific dialect words, catchphrases), controlling the syntactic structure complexity, matching the habitual emoji or punctuation style, etc., so that the final output text is highly consistent in style with the target person.

[0066] Method 3 Step S129: Generate virtual appearance and voice sound data corresponding to the personalized answer content through the appearance and voice model.

[0067] The above appearance and voice model can be obtained by training the image data and audio data in the multi-modal data, and the virtual 2D character or virtual 3D character on the image model realizes the simulation of the virtual appearance and the cloning of the voice.

[0068] It can be understood that generating virtual appearance and voice sound data corresponding to the personalized answer content through the appearance and voice model can realize the multi-modal output capability of the dialogue system, and expand the text answer to a comprehensive interactive experience containing visual and auditory elements. This technical effect is derived from the ability to map the characteristics of the personality model to multi-modal forms, enabling the system to further strengthen personalized expression through virtual appearance and voice characteristics, and improve the immersion and affinity of the interaction.

[0069] In specific implementation, the above three methods can be applied alternatively or in combination (two combinations or three combinations).

[0070] As an optional implementation of the above intelligent interaction method, after sending the personalized answer content to the terminal device, it further includes: Step S140: Receive the user feedback data sent by the terminal device, which is fed back by the corresponding user of the terminal device according to the personalized answer content.

[0071] The above electronic device can also receive user feedback data sent by the terminal device, which can include user like data, user dislike data, text correction data and other new interactive data. These new interactive data can be used for incremental training of each sub-model in the personality model.

[0072] In a specific implementation, the electronic device receives user feedback data from the terminal device (such as a mobile phone App, a webpage, or a smart speaker) in real time through the interaction interface module, which is generated by the user after viewing or listening to the personalized answer. The feedback forms include explicit behaviors (such as clicking the "like" or "dislike" button), semi-structured inputs (such as selecting "answer inaccurate" or "tone inappropriate" labels), and free text corrections (such as the user directly editing and submitting a more expected answer version, for example, changing "you were naughty as a child" to "you were naughty as a child but had strong opinions"). In addition, the electronic device can also collect implicit feedback signals such as user dwell time, whether to continue to ask questions, changes in voice tone, etc. All these new interaction data are structured and stored in the feedback database, with the associated original answer, conversation context, and timestamp, as training samples for subsequent model optimization.

[0073] Step S150: updating the personal knowledge base according to the user feedback data to obtain an updated personal knowledge base.

[0074] In a specific implementation, the continuous learning and updating module automatically analyzes the user feedback data collected in step S140: if the feedback is "dislike" with text correction, the system extracts the factual information in the correction content (such as event details, point of view), which is written into the personal knowledge base as a new or corrected item after credibility verification (such as consistency comparison with preset user historical data). If the feedback is "like", the confidence weight or frequency score of the corresponding knowledge item is increased; for emotional feedback (such as the user pointing out "tone too cold"), the system does not directly modify the knowledge base facts, but records the deficiencies of the knowledge item associated with the answer in emotional expression for language style model adjustment reference; the update process uses an incremental writing mechanism to ensure that the knowledge base always remains up-to-date and consistent, forming a dynamic evolving personal memory system.

[0075] In the implementation process of the above scheme, by receiving user feedback data and updating the personal knowledge base according to the same, the dynamic optimization of the knowledge base is realized, so that the system can continuously adapt to the changes in user needs, thereby improving the real-time and accuracy of the answer content. This closed-loop feedback mechanism ensures the self-evolution ability of the knowledge base, avoiding the lag or outdated problems caused by static knowledge base.

[0076] Step S160: incrementally training the personality model based on the updated personal knowledge base.

[0077] It can be understood that the above-mentioned personality model is not a static model, because after incrementally training the personality model based on the updated personal knowledge base, the personality model can be continuously trained and continuously learned, and can continue to develop into a more in-depth and emotionally interactive digital twin model.

[0078] In a specific implementation, the system triggers the incremental training process periodically or after accumulating sufficient feedback data: first, high-quality question-answer pairs or structured knowledge fragments are sampled from the updated personal knowledge base to construct a fine-tuning dataset. Subsequently, lightweight training is performed for different sub-modules of the personality model, for example, parameter fine-tuning of the personality model using new knowledge entries, optimization of the word frequency distribution and sentence generation strategy of the language style model using language correction samples, and adjustment of the emotional response threshold of the personality emotion model combined with emotional feedback. The training process uses efficient methods such as Low Rank Adaptation (LoRA) or Prompt Tuning to avoid full-parameter retraining, thereby quickly absorbing new feedback while maintaining the original capabilities. Finally, the updated sub-models are integrated into the AI dialogue engine, enabling the digital twin to exhibit more accurate and user-desired personalized behavior in subsequent interactions.

[0079] As an optional implementation of the above intelligent interaction method, the pre-set user and the corresponding user of the terminal device are in a kinship relationship, an identity continuation relationship, an emotional support relationship, or an idealized relationship. It can be understood that these relationship limitations enable the system to build more natural interaction logic based on specific social or emotional bonds, for example, a kinship relationship may involve more caring language and behavior, while an idealized relationship may focus more on motivation and positive feedback, thereby enhancing the emotional resonance and interaction experience of the user. Further, by explicitly defining the relationship type between the pre-set user and the corresponding user, the system can more efficiently filter irrelevant interaction scenarios, reduce waste of computing resources, and improve response speed, as the system only needs to optimize algorithms and data processing procedures for specific relationship types.

[0080] See Figure 2 The structure of the intelligent interaction device provided by the embodiment of the application is shown. The embodiment of the application provides an intelligent interaction device 200, which comprises: The conversation data receiving module 210 is configured to receive the current conversation data sent by the terminal device.

[0081] The answer content generating module 220 is configured to generate personalized answer content for the current conversation data based on the personality model created for the pre-set user, and the personality model is constructed based on the multi-modal data of the pre-set user, and the pre-set user is an object simulated by the corresponding user of the terminal device.

[0082] The answer content sending module 230 is configured to send the personalized answer content to the terminal device.

[0083] As an optional implementation of the above device, the intelligent interaction device further comprises: The modal data receiving module is configured to receive the multi-modal data of the pre-set user.

[0084] a structured information extraction module configured to extract structured information from the multi-modal data.

[0085] a personality model construction module configured to construct a personality model based on the structured information.

[0086] a personality model calling module configured to enable the base model to call the personality model to generate personalized answer content by fine-tuning and prompting the base model.

[0087] In an optional embodiment of the above device, the personality model comprises a personal knowledge base, and the answer content generation module comprises: an associated information extraction submodule configured to extract associated information from the current session data and historical session data of the terminal device.

[0088] a session intent recognition submodule configured to recognize a session intent of the current session data according to the personality model and the associated information.

[0089] a related information retrieval submodule configured to retrieve related information from the personal knowledge base based on the session intent.

[0090] an answer content generation submodule configured to generate personalized answer content according to the retrieval result.

[0091] In an optional embodiment of the above device, the answer content generation module comprises: an embedded representation obtaining submodule configured to, if the current session data contains multi-modal content, obtain an embedded representation vector by embedding encoding the multi-modal content by a multi-modal encoder in the AI dialogue engine, the multi-modal encoder comprising a text encoder, an image encoder, an audio encoder, and / or a video encoder.

[0092] an answer content generation submodule configured to generate personalized answer content for the current session data according to the embedded representation vector by the personality model.

[0093] In an optional embodiment of the above device, the personality model further comprises a personality-emotion model, a language style model, and / or an appearance-sound model, and the intelligent interactive device further comprises: a state tendency adjustment module configured to analyze an emotional state of the current session data by the personality-emotion model, and adjust an emotional tendency of the personalized answer content according to the emotional state.

[0094] and / or a language style control module configured to analyze a language style of the current session data by the language style model, and control a word usage habit, a speech speed feature, or a dialect proportion of the personalized answer content according to the language style.

[0095] And / or, the image data generation module is configured to generate a virtual appearance image and voice sound data corresponding to the personalized answer content through an appearance sound model.

[0096] As an optional implementation of the above device, the intelligent interaction device further comprises: The feedback data receiving module is configured to receive user feedback data sent by the terminal device, the user feedback data being feedback by a corresponding user of the terminal device according to the personalized answer content.

[0097] The personal knowledge base updating module is configured to update the personal knowledge base according to the user feedback data to obtain an updated personal knowledge base.

[0098] The personality model training module is configured to perform incremental training on the personality model based on the updated personal knowledge base.

[0099] As an optional implementation of the above device, the corresponding user of the preset user and the terminal device is a kinship, an identity continuation relationship, an emotional support relationship or an idealized relationship.

[0100] It should be understood that the device corresponds to the above-mentioned intelligent interaction method embodiments, and can perform various steps involved in the above-mentioned method embodiments. The specific functions of the device can be referred to the description in the above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0101] Please refer to Figure 3 The electronic device 300 provided by the embodiment of the present application includes a processor 310 and a memory 320. The memory 320 stores machine readable instructions executable by the processor 310. When the machine readable instructions are executed by the processor 310, the method described above is performed.

[0102] The embodiment of the present application further provides a computer readable storage medium 330, which stores a computer program. The computer program is run by the processor 310 to execute the method described above. The computer readable storage medium 330 can be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0103] The embodiment of the present application further provides a computer program product, which comprises a computer program or computer instruction. The computer program or computer instruction is run by the processor to execute the method described above.

[0104] It should be noted that each embodiment in the specification adopts a progressive manner for description. Each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. The relevant part can be referred to the part of the method embodiment.

[0105] In several embodiments provided by the embodiment of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other ways. The apparatus embodiment described above is only schematic. For example, the flow chart and block diagram in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flow chart or block diagram can represent a module, program segment or part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can be performed in a different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved.

[0106] In addition, each function module in each embodiment in the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0107] The above description is only an optional implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. An intelligent interaction method, characterized in that, The method comprises the following steps: receiving current session data sent by a terminal device; generating personalized answer content for the current session data based on a personality model created for a preset user, the personality model being constructed based on multi-modal data of the preset user, the preset user being an object intended to be simulated by a corresponding user of the terminal device; sending the personalized answer content to the terminal device.

2. The method of claim 1, wherein, Before the step of generating personalized answer content for the current session data based on a personality model created for a preset user, the method further comprises the following steps: receiving multi-modal data of the preset user; extracting structured information from the multi-modal data; constructing the personality model based on the structured information; fine-tuning and prompting an engineering unit configuration base model to enable it to call the personality model to generate the personalized answer content.

3. The method of claim 2, wherein, The personality model comprises a personal knowledge base; the step of generating personalized answer content for the current session data comprises the following steps: extracting associated information from the current session data and historical session data of the terminal device; identifying a session intent of the current session data according to the personality model and the associated information; retrieving relevant information from the personal knowledge base based on the session intent; generating the personalized answer content according to the retrieval result.

4. The method of claim 2, wherein, The step of generating personalized answer content for the current session data comprises the following steps: if the current session data contains multi-modal content, embedding and encoding the multi-modal content by a multi-modal encoder in an AI dialogue engine to obtain an embedding representation vector, the multi-modal encoder comprising a text encoder, an image encoder, an audio encoder and / or a video encoder; generating personalized answer content for the current session data according to the embedding representation vector by the personality model.

5. The method of claim 3, wherein, The personality model further comprises a personality emotion model, a language style model and / or an appearance and voice model; after the step of generating personalized answer content for the current session data based on a personality model created for a preset user, the method further comprises the following steps: analyzing an emotional state of the current session data by the personality emotion model, and adjusting an emotional tendency of the personalized answer content according to the emotional state; and / or, analyzing a language style of the current session data by the language style model, and controlling a word usage habit, a speech speed feature or a dialect proportion of the personalized answer content according to the language style; and / or, generating a virtual appearance image and voice data corresponding to the personalized answer content by the appearance and voice model.

6. The method of claim 5, wherein, After the step of sending the personalized answer content to the terminal device, the method further comprises the following steps: receiving user feedback data sent by the terminal device, the user feedback data being feedback by a corresponding user of the terminal device according to the personalized answer content; updating the personal knowledge base according to the user feedback data to obtain an updated personal knowledge base; incrementally training the personality model based on the updated personal knowledge base.

7. The method of claim 1, wherein, The preset user and the corresponding user of the terminal device are in a kinship relationship, an identity continuation relationship, an emotional support relationship or an idealized relationship.

8. An intelligent interactive device, characterized by The method comprises the following steps: The session data receiving module is configured to receive current session data sent by a terminal device. The answer content generating module is configured to generate personalized answer content for the current session data based on a personality model created for a preset user, the personality model being constructed based on multi-modal data of the preset user, the preset user being an object intended to be simulated by a corresponding user of the terminal device. The answer content sending module is configured to send the personalized answer content to the terminal device.

9. An electronic device, comprising: The processor and the memory, wherein the memory stores machine readable instructions executable by the processor, and the machine readable instructions, when executed by the processor, perform the method of any one of claims 1 to 7. The computer program or computer instructions, wherein the computer program or the computer instructions, when executed by the processor, perform the method of any one of claims 1 to 7.

10. A computer program product, characterised in that, ​ ​

Images