Human-computer interaction method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention discloses a man-machine interaction method and device, electronic equipment, a storage medium and a program, and the method comprises the steps: obtaining current interaction request data inputted by a target user; retrieving a local knowledge base according to current interaction request data input by the target user to obtain target historical knowledge question and answer data corresponding to the current interaction request data; generating large model interaction data according to the target historical knowledge question and answer data and the current interaction request data; and requesting a target large model according to the large model interaction data to generate current interaction response data of the current interaction request data. According to the technical scheme of the embodiment of the invention, the safety of user data can be guaranteed, and the safety and reliability of a man-machine interaction process are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of human-computer interaction technology, and in particular to a human-computer interaction method, device, electronic device, storage medium and program. Background Technology

[0002] Retrieval-Augmented Generation (RAG) technology, as a typical enhancement architecture for large language models, can effectively alleviate the problem of insufficient coverage of private domain knowledge in large language models. The core execution logic of this architecture is as follows: before generating the target response content, information related to the current interaction is retrieved from a pre-set knowledge base, and this information is used as context input to the large language model, so that the content output by the model can better match the user's personalized needs, the operating logic of specific devices, or the business rules of specific application scenarios.

[0003] However, traditional RAG systems generally adopt a cloud-based deployment architecture, where the knowledge base and corresponding search services are deployed on cloud servers. In the actual operation of this architecture, user-initiated query requests and core private data, including personalized configurations and private business data, must be uploaded to the cloud server for processing. During this process, the aforementioned data faces a high probability of security risks, including leakage, misuse by service providers, or unauthorized access by malicious entities. Summary of the Invention

[0004] This invention provides a human-computer interaction method, device, electronic device, storage medium, and program that can ensure user data security, thereby improving the security and reliability of the human-computer interaction process.

[0005] According to one aspect of the present invention, a human-computer interaction method is provided, comprising:

[0006] Obtain the current interaction request data input by the target user;

[0007] The local knowledge base is retrieved based on the current interaction request data input by the target user to obtain the target historical knowledge question and answer data corresponding to the current interaction request data;

[0008] Generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data;

[0009] Based on the large model interaction data request target large model, the current interaction response data of the current interaction request data is generated.

[0010] According to another aspect of the present invention, a human-computer interaction device is provided, comprising:

[0011] The current interaction request data acquisition module is used to acquire the current interaction request data input by the target user;

[0012] The target historical knowledge question and answer data determination module is used to retrieve the target historical knowledge question and answer data corresponding to the current interaction request data input by the target user from the local knowledge base.

[0013] The large model interaction data generation module is used to generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data.

[0014] The current interaction response data generation module is used to generate current interaction response data of the current interaction request data based on the target large model of the large model interaction data request.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the human-computer interaction method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the human-computer interaction method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the human-computer interaction method described in any embodiment of the present invention.

[0021] This invention acquires the current interaction request data input by the target user and retrieves the local knowledge base based on this data to obtain the target historical knowledge question-and-answer data corresponding to the current interaction request data. Further, it generates large-scale model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data. After obtaining the large-scale model interaction data, it requests the target large-scale model to generate the current interaction response data for the current interaction request data. This solution addresses the data leakage and privacy security risks present in existing RAG systems during data processing, ensuring user data security and thus improving the security and reliability of the human-computer interaction process.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a human-computer interaction method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a human-computer interaction method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a flowchart illustrating a human-computer interaction method based on an audio device provided in Embodiment 2 of the present invention;

[0027] Figure 4 This is a flowchart illustrating a personalized hybrid retrieval and recall method provided in Embodiment 2 of the present invention;

[0028] Figure 5 This is a flowchart of a reasoning question-and-answer method provided in Embodiment 2 of the present invention;

[0029] Figure 6 This is a flowchart of a data storage and archiving method provided in Embodiment 2 of the present invention;

[0030] Figure 7 This is a schematic diagram of a human-computer interaction device provided in Embodiment 3 of the present invention;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," and "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Example 1

[0035] Figure 1 This is a flowchart of a human-computer interaction method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the knowledge base relied upon in the human-computer interaction process is deployed in a local storage device. The method can be executed by a human-computer interaction device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a terminal device or a server device, as long as it can execute the human-computer interaction method. The present invention does not limit the specific type of electronic device. Correspondingly, as... Figure 1 As shown, the method includes the following operations:

[0036] S110. Obtain the current interaction request data input by the target user.

[0037] The target user can be the user currently interacting with the device. The current interaction request data can be the request data initiated by the target user for a specific need during the real-time session of the human-computer interaction, triggering the business system to execute the corresponding processing flow.

[0038] In this embodiment of the invention, users who have human-computer interaction needs and wish to establish a session connection with the RAG system can be designated as target users. During the real-time session of human-computer interaction, target users can flexibly use text input or voice input methods according to their actual business scenario operating habits to actively submit current interaction request data to the RAG system, so that the RAG system can initiate the corresponding processing flow upon receiving the current interaction request data.

[0039] S120. Retrieve the local knowledge base based on the current interaction request data input by the target user to obtain the target historical knowledge question and answer data corresponding to the current interaction request data.

[0040] The local knowledge base can be a collection of personalized private data of the target user, deployed on the target user's local electronic device. For example, the local knowledge base can be deployed on a local electronic device such as a speaker or headphones. In a specific example, the local knowledge base can be used to store historical knowledge question-and-answer data, which may include, but is not limited to, historical interaction request data, historical interaction response data corresponding to the historical interaction request data, and historical question-and-answer feedback information recorded for each historical interaction response data, as well as its corresponding interaction weight. The target historical knowledge question-and-answer data can be historical knowledge question-and-answer data in the local knowledge base that is highly correlated with the target user's current interaction request data.

[0041] Accordingly, after acquiring the current interaction request data input by the target user, the RAG system can parse this data to determine the target user's actual interaction needs and generate a standardized retrieval vector. Furthermore, based on the standardized retrieval vector generated from the current interaction request data, the RAG system can retrieve historical knowledge question-and-answer data stored in the local knowledge base, filtering out target historical knowledge question-and-answer data from the local knowledge base that has a high correlation with the current interaction request data.

[0042] S130. Generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data.

[0043] Among them, the large model interaction data can be the interaction data of the input large model generated based on the target's historical knowledge question and answer data and the current interaction request data.

[0044] Correspondingly, after filtering the target historical knowledge question and answer data corresponding to the current interaction request data from the local knowledge base, the RAG system can combine the target historical knowledge question and answer data with the current interaction request data to generate large model interaction data that can accurately match the target user's real interaction needs and highly fit the target user's historical interaction preferences.

[0045] S140. Generate the current interaction response data of the current interaction request data based on the target large model according to the large model interaction data request.

[0046] The target large model can be any large language model that performs human-computer interaction. The current interaction response data can be the response data generated by the target large model based on the interaction data of the large model, corresponding to the current interaction request data.

[0047] Correspondingly, after obtaining the large model interaction data, the large model interaction data can be input into the target large model so that the target large model can generate current interaction response data that is highly consistent with the target user's current interaction request data in terms of core intent, content description, and format requirements.

[0048] The target large language model can be any available large language model (LLM), used to generate the current interaction response data corresponding to the current interaction request data based on the large model's interaction data. Large language models, also called large-scale language models, refer to deep learning models trained with a large amount of relevant data (such as text data, speech data, or combined text and image data) capable of processing text sequences. They can generate natural language text or understand the meaning of language text; these models typically have billions of parameters. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and have wide applications. The input to a large language model is data, such as text data, speech data, or combined text and image data. The large language model encodes the input data to obtain corresponding word vector representations, and then decodes the encoded word vectors to automatically complete the input data processing and obtain the corresponding output data. For example, text can be input into a large language model, which will process and predict the input text, outputting the corresponding response text.

[0049] Therefore, the human-computer interaction method provided in this invention deploys a knowledge base storing historical knowledge question-and-answer data on the target user's local device. This effectively protects the privacy and integrity of user data by avoiding potential security risks such as interception, theft, and leakage during network transmission and storage on remote servers. Furthermore, the locally deployed knowledge base grants users complete autonomy in management, allowing them to flexibly supplement, precisely filter, and iteratively optimize data according to their needs. Compared to the traditional model that relies on unified updates from a remote server, this invention's local iteration mechanism significantly improves the iteration efficiency and flexibility of the knowledge base and enables real-time responses to dynamic changes in users' personalized needs, thereby significantly enhancing the user adaptability and practical value of the human-computer interaction method.

[0050] This invention acquires the current interaction request data input by the target user and retrieves the local knowledge base based on this data to obtain the target historical knowledge question-and-answer data corresponding to the current interaction request data. Further, it generates large-scale model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data. After obtaining the large-scale model interaction data, it requests the target large-scale model to generate the current interaction response data for the current interaction request data. This solution addresses the data leakage and privacy security risks present in existing RAG systems during data processing, ensuring user data security and thus improving the security and reliability of the human-computer interaction process.

[0051] Example 2

[0052] Figure 2 This is a flowchart of a human-computer interaction method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and is further specified. In this embodiment, specific optional implementation methods are given for retrieving the local knowledge base according to the current interaction request data input by the target user to obtain the target historical knowledge question and answer data corresponding to the current interaction request data, and for requesting the target large model to generate the current interaction response data of the current interaction request data based on the interaction data of the large model. Accordingly, as Figure 2 As shown, the method in this embodiment may include:

[0053] S210. Obtain the current interaction request data input by the target user.

[0054] S220. Extract multimodal features from the current interaction request data to obtain the first multimodal fusion feature vector of the current interaction request data.

[0055] The first multimodal fusion feature vector can be a feature vector obtained by extracting multimodal features from the current interactive request data.

[0056] Specifically, after obtaining the current interaction request data input by the target user, multimodal features can be extracted from the current interaction request data input by the target user, and the extracted multimodal features can be fused to obtain the first multimodal fused feature vector of the current interaction request data.

[0057] In an optional embodiment of the present invention, if the current interaction request data is voice interaction request data, then the step of extracting multimodal features from the current interaction request data to obtain a first multimodal fusion feature vector of the current interaction request data may include: converting the current interaction request data into text interaction request data; extracting text features from the text interaction request data to obtain text semantic features of the current interaction request data; extracting audio features from the current interaction request data to obtain audio semantic features of the current interaction request data; wherein the audio semantic features include at least one of emotion features, intent features, entity features, and speech features; and concatenating the text semantic features and the audio semantic features to obtain a first multimodal fusion feature vector of the current interaction request data.

[0058] The voice interaction request data can be interaction request data input by the target user in voice form. The text interaction request data can be data obtained by converting the current interaction request data into text form. The text semantic features can be features obtained by extracting text features from the text interaction request data. The audio semantic features can be features obtained by extracting audio features from the current interaction request data. For example, audio semantic features can include, but are not limited to, emotion features, intent features, entity features, and voice features, etc., and this embodiment of the invention does not limit the specific types included in the audio semantic features.

[0059] Figure 3 This is a flowchart illustrating a human-computer interaction method based on an audio device according to Embodiment 2 of the present invention. Specifically, as shown... Figure 3As shown, target users can submit interaction requests to the PH-RAG (Personalized Hybrid Retrieval-Augmented Generation) system via voice questions, either through the system's built-in speakers or external audio devices connected to the PH-RAG system. After receiving the voice interaction request data from the target user, the PH-RAG system can convert the voice interaction request data into text questions, i.e., text interaction request data, using an ASR (Automatic Speech Recognition) model. For example, ASR models can include, but are not limited to, Whisper (Weakly Supervised Hyperscale Intelligent Speech Recognition) and Conformer (Convolution-augmented Transformer). Furthermore, textual vectorization models such as Sentence-BERT (Sentence Bidirectional Encoder Representations from Transformers) or BGE Embedding (Bilingual General Embedding) can be used to vectorize the textual questions, obtaining the textual semantic features of the current interactive request data input by the target user. Simultaneously, the PH-RAG system can obtain the frequency domain information of the current interactive request data using techniques such as Mel (Mel Spectrum) or MFCC (Mel-Frequency Cepstral Coefficients). Further, algorithms such as Whisper, Hubert (Hierarchical Universal BERT for Embedding Representation), or Wav2vec (Waveform to Vector) can be used to extract the audio semantic features of the current interactive request data based on its frequency domain information. After obtaining the text semantic features and audio semantic features of the current interactive request data, the text semantic features and audio semantic features can be concatenated to obtain the first multimodal fusion feature vector of the current interactive request data.

[0060] In a specific example, emotion recognition can be performed on the current interaction request data input by the target user to predict the target user's emotional state. To more accurately recall historical question-and-answer data, intent recognition can also be performed on the current interaction request data to predict the target user's interaction intent, such as playing music, checking the weather, or setting an alarm. This embodiment of the invention does not limit the target user's interaction intent. After intent recognition, entity information of the user interaction can be obtained through slot filling. Simultaneously, algorithms such as Hubert or Wav2vec can be used to obtain the target user's voice features based on the current interaction request data input by the target user, such as rapid sounds, applause, content style, and background noise.

[0061] In a specific example, suppose a target user initiates a voice interaction request to the PH-RAG system in an agitated state, with the voice message "Tell me a bedtime story." The PH-RAG system can use an ASR model to convert the user's voice interaction request into a text interaction request, namely, "Tell me a bedtime story." Simultaneously, the PH-RAG system can also extract and analyze features from the user's voice interaction request using an intent recognition model, identifying the target user's intent as "tell a story," and can also determine the target user's emotion as "agitated" using an audio model. Based on the above, the PH-RAG system can construct a fused feature vector including the text feature: "Tell me a bedtime story," the emotion feature: "agitated," and the intent feature: "tell a story."

[0062] In an optional embodiment of the present invention, the step of extracting text features from the text interaction request data to obtain the text semantic features of the current interaction request data may include: performing word segmentation on the text interaction request data using a preset word segmenter to obtain the word segmentation result of the text interaction request data; and performing text feature extraction on the word segmentation result of the text interaction request data using a text vector model to obtain the text semantic features of the current interaction request data.

[0063] The preset word segmenter can be a word segmenter with predefined word segmentation rules and logic, such as, but not limited to, the jieba word segmenter. The word segmentation result can be the result obtained by segmenting text interaction request data. The text vector model can be a model that converts unstructured text data into structured numerical vectors.

[0064] In this embodiment of the invention, when extracting text features from text interaction request data to obtain the text semantic features of the current interaction request data, the word segmenter can first be optimized according to the human-computer interaction scenario to obtain a preset word segmenter. Further, the preset word segmenter can be used to perform word segmentation processing on the text interaction request data to obtain the word segmentation processing result, that is, to convert the text interaction request data into tokens. Further, a text vector model can be used to extract text features from the word segmentation processing result of the text interaction request data, thereby obtaining the text semantic features of the current interaction request data.

[0065] In an optional embodiment of the present invention, the step of extracting audio features from the current interaction request data to obtain the audio semantic features of the current interaction request data may include: extracting frequency domain information of the current interaction request data; and extracting audio features from the frequency domain information of the current interaction request data to obtain the audio semantic features of the current interaction request data.

[0066] Specifically, when extracting audio features from the current interaction request data to obtain its audio semantic features, the current interaction request data can first be converted from continuous time-domain information to frequency-domain information, which makes it easier to extract audio features. Furthermore, audio features can be extracted based on the frequency-domain information of the current interaction request data to obtain its audio semantic features.

[0067] Continuing with the example above, the text semantic features and audio semantic features extracted from the current interaction request data can be represented as: {“tokens”:[964, 372, 53, 34, 974, 1092, 7301, 287, 342, 92, 1753, 2740, 982]; “text_vector”:[0.2321, 0.1092, ..., 0.9832, 0.0821]; “speech_vector”:[0.098, 0.872, ..., 0.102, 0.897]}. Here, `tokens` is the word sequence obtained after tokenizing the text interaction request data; `text_vector` is the text semantic feature of the current interaction request data obtained by using a text vector model after tokenizing the text interaction request data; `speech_vector` is the audio semantic feature corresponding to the current interaction request data. The closer its value is to 0, the lower the degree of inclusion of the corresponding semantic feature; the closer the value is to 1, the higher the degree of inclusion of the corresponding semantic feature. In the example above, the feature value of the storytelling intention is 0.872, the feature value of the emotion of agitation is 0.897, and the feature values ​​of the other semantic features are all close to 0.

[0068] Furthermore, the text semantic features and audio semantic features extracted from the current interaction request data can be fused to obtain the first multimodal fusion feature vector of the current interaction request data. For example, the first multimodal fusion feature vector can be represented as {“tokens”:[964,372,53,34,974,1092,7301,287,342,92,1753,2740,982]; “vector”:[0.2321,0.1092,...,0.9832,0.0821,0.098,0.872,...,0.102,0.897]}.

[0069] S230. Extract multimodal features from each historical knowledge question and answer data in the local knowledge base to obtain the second multimodal fusion feature vector of each historical knowledge question and answer data.

[0070] The second multimodal fusion feature vector can be a feature vector obtained by extracting multimodal features from historical knowledge question-and-answer data and fusing the extracted multimodal features.

[0071] Specifically, during human-computer interaction, the PH-RAG system can extract multimodal features from all historical knowledge question-and-answer data stored in the local knowledge base. Furthermore, the extracted multimodal features can be fused to obtain a second multimodal fused feature vector for each historical knowledge question-and-answer data point.

[0072] In an optional embodiment of the present invention, the step of extracting multimodal features from each historical knowledge question-and-answer data in the local knowledge base to obtain a second multimodal fusion feature vector for each historical knowledge question-and-answer data may include: extracting key information features and semantic information features from the question-and-answer interaction data in the historical knowledge question-and-answer data; extracting feedback information features from the answer feedback data in the historical knowledge question-and-answer data; and fusing the key information features, the semantic information features, and the feedback information features to obtain a second multimodal fusion feature vector for the historical knowledge question-and-answer data.

[0073] The question-and-answer interaction data can be a collective term for historical interaction request data and corresponding historical interaction response data within the historical knowledge question-and-answer data. Key information features can be features obtained by extracting keywords from the question-and-answer interaction data in the historical knowledge question-and-answer data. Semantic information features can be features obtained by extracting text features from the question-and-answer interaction data in the historical knowledge question-and-answer data. Answer feedback data can be a collective term for the historical question-and-answer feedback information recorded for each historical interaction response data, and its corresponding weight parameters. Feedback information features can be features obtained by extracting features from the answer feedback data in the historical knowledge question-and-answer data.

[0074] Figure 4 This is a flowchart illustrating a personalized hybrid retrieval and recall method provided in Embodiment 2 of the present invention. Specifically, as shown... Figure 3 and Figure 4 As shown, after obtaining the first multimodal fusion feature vector of the current interaction request data, a PH recall operation can be performed in the local knowledge base based on the first multimodal fusion feature vector. Specifically, keywords from the question-and-answer interaction data in historical knowledge question-and-answer data can be extracted using techniques such as jieba word segmentation, and the key information features of each historical knowledge question-and-answer data can be calculated using the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm. Simultaneously, the Sentence-BERT (Sentence Bidirectional Encoder Representations from Transformers) algorithm can be used for text encoding to extract semantic information features from question-and-answer interaction data in historical knowledge question-and-answer data. Furthermore, feedback information features can be extracted from the answer feedback data in historical knowledge question-and-answer data. Furthermore, key information features, semantic information features, and feedback information features can be fused to obtain a second multimodal fusion feature vector of historical knowledge question-answering data. and .

[0075] S240. Calculate the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector.

[0076] Among them, the retrieval recall score can be used as an evaluation metric to measure the degree of correlation between current interaction request data and historical knowledge question and answer data.

[0077] Specifically, after obtaining the first multimodal fusion feature vector and the second multimodal fusion feature vector, a retrieval recall score can be calculated between the first multimodal fusion feature vector and the second multimodal fusion feature vector to determine the degree of correlation between the current interactive request data and each historical knowledge question and answer data based on the aforementioned retrieval recall score.

[0078] Continuing with the example above, as follows: Figure 4 As shown, the second multimodal fusion feature vector of historical knowledge question-and-answer data is obtained. and Subsequently, an attention mechanism can be used to analyze the first multimodal fusion feature vector Q and the second multimodal fusion feature vector. and Interactive features are extracted, and the extracted multidimensional fusion features are mapped to score values ​​ranging from [0, 1] using an FNN (Feedforward Neural Network). .

[0079] In an optional embodiment of the present invention, calculating the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector may include: calculating the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector based on the following formula:

[0080] ;

[0081] ;

[0082] ;

[0083] in, The key vector matrix of the historical knowledge question-and-answer data. Let be the value vector matrix of the historical knowledge question and answer data. and Together they constitute the second multimodal fusion feature vector. The weight matrix applicable to the key vector matrix. The weight matrix to which the value vector matrix is ​​applied. These are the key information features of the question-and-answer interaction data in the historical knowledge question-and-answer data. The semantic information features of the question-and-answer interaction data in the historical knowledge question-and-answer data, The feedback information features of the answer feedback data in the historical knowledge question-and-answer data. For concatenation functions, The retrieval recall score is given. It is a feedforward neural network. For the attention calculation function, This is the first multimodal fusion feature vector of the current interactive request data.

[0084] S250. Based on the retrieval recall score, filter the target historical knowledge question and answer data from each of the historical knowledge question and answer data.

[0085] Specifically, after calculating the retrieval recall score between the first and second multimodal fusion feature vectors, the correlation between the current interaction request data and each historical knowledge question-and-answer data can be determined based on this score. Furthermore, k target historical knowledge question-and-answer data with high correlation to the current interaction request data can be selected from the historical knowledge question-and-answer data.

[0086] S260. Generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data.

[0087] In an optional embodiment of the present invention, generating large model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data may include: determining a large model interaction prompt template; and filling the large model interaction prompt template with the target historical knowledge question-and-answer data and the current interaction request data to obtain the large model interaction data.

[0088] The large model interaction prompt template can be a predefined, standardized template of large model interaction data. The large model interaction data can be determined based on the target's historical knowledge question-and-answer data and current interaction request data, and is the standardized data input to the target large model.

[0089] In this embodiment of the invention, when generating large-scale model interaction data based on target historical knowledge question-and-answer data and current interaction request data, a large-scale model interaction prompt template can first be predefined to unify the input data format of the target large-scale model and ensure the standardization and consistency of the input data. Furthermore, raw data to be input into the target large-scale model can be constructed based on the target historical knowledge question-and-answer data and current interaction request data. This raw data can then be substituted into the predefined large-scale model interaction prompt template for format standardization processing, ultimately generating large-scale model interaction data that meets the input requirements of the target large-scale model.

[0090] Continuing with the example above, after recalling the tokens of the top-k target historical knowledge question-and-answer data based on the retrieval recall score, the returned top-k target historical knowledge question-and-answer data can be detokenized. For example, after detokenizing the target historical knowledge question-and-answer data, it can be represented as {"token": [64, 1272, 663, 832, 90, 192, 5301, 187, 242, 12, 753, 2140, 182]; "question": "Tell me a story about a rabbit and a tortoise, about 500 words long, that a 4-year-old would enjoy."; "answer": "Once upon a time, in a beautiful forest, lived a hopping little rabbit named Bai Bai, and a slowly crawling little tortoise named Man Man..."}. Simultaneously, based on historical information, we understand the target user profile is "a mother around 30 years old who enjoys listening to bedtime stories about small animals," and that "the stories are approximately 5 minutes long." Therefore, this information can be combined into a prompt and sent to the target large model as interaction data. For example, the large model interaction data could be:

[0091] "Please answer the user's question based on the context information provided below. If the context information is insufficient to answer the question, please simply state that you do not know; do not fabricate an answer."

[0092] Context information:

[0093] Question: Please tell me a story about the race between the rabbit and the tortoise, about 500 words long, that my 4-year-old child would enjoy listening to. It's a story that can soothe their mood.

[0094] Answer: Once upon a time, in a beautiful forest, lived a hopping little rabbit named Bai Bai, and a slowly crawling little turtle named Man Man. ...}

[0095] User question: {Tell me a bedtime story.}

[0096] User's mood: Frustrated.

[0097] User profile: A stay-at-home mom with a 4-year-old child.

[0098] Context-based answer:

[0099] S270. Generate text interaction response data for the current interaction request data based on the target large model according to the large model interaction data request.

[0100] Among them, text interaction response data can be text-type response data generated by the target large model based on the interaction data request of the large model.

[0101] like Figure 3 As shown, after obtaining the large model interaction data, the PH-RAG system can request the target large model based on the large model interaction data. After receiving the model interaction data, the target large model can generate text interaction response data for the current interaction request data based on the large model interaction data.

[0102] S280. The text interaction response data is converted into the current interaction response data of the current interaction request data through a text-to-speech model.

[0103] Accordingly, after obtaining the text-based interaction response data of the current interaction request, the PH-RAG system can use a TTS (Text-To-Speech) model to convert the text-based interaction response data into speech, generating the current interaction response data in speech form consistent with the target voiceprint features. Furthermore, the PH-RAG system can play the speech-based current interaction response data to the target user in real time through external or built-in audio devices.

[0104] In an optional embodiment of the present invention, after generating the current interaction response data of the current interaction request data based on the target large model interaction data request according to the large model interaction data, the method may further include: obtaining the target user's current question-and-answer feedback information on the current interaction response data; calculating the interaction weight of the current question-and-answer feedback information based on the parsing result of the current question-and-answer feedback information; and storing the current interaction request data, the current interaction response data, the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information in the local knowledge base.

[0105] The current question-and-answer feedback information can be the relevant information returned by the target user after receiving the current interaction response data. The interaction weight of the current question-and-answer feedback information can be a metric assigned to the current question-and-answer feedback information to quantify its impact on subsequent model input data construction and interaction strategy adjustments.

[0106] In this embodiment of the invention, after the target large model generates the current interaction response data based on the interaction data request of the large model, the PH-RAG system can collect relevant information fed back by the target user after receiving the current interaction response data, namely the current question-and-answer feedback information. Further, the current question-and-answer feedback information can be parsed, and the target user's satisfaction with the current interaction response data can be determined based on the parsing results. The interaction weight of the current question-and-answer feedback information can also be quantitatively calculated based on the parsing results. On this basis, the current interaction request data, current interaction response data, current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information can be stored in a local knowledge base. The above solution, through continuous recording, parsing, and quantitative analysis of the user's full-process interaction behavior, can achieve dynamic construction and iterative optimization of the target user's personal profile and interaction preferences, providing data support for the precise adjustment of subsequent interaction strategies. For example, emotional expressions such as laughter, sighs, anger, concise answers, or low-density information from the target user can be used as current question-and-answer feedback information.

[0107] Continuing with the example above, if the target user's actual need is not to obtain the "Tortoise and the Hare" themed story generated by the target model, and the target user provides feedback such as "I don't want to hear the Tortoise and the Hare story anymore" after receiving the generated content on that theme, the PH-RAG system can assign a lower weight to the current question-and-answer feedback information generated in this interaction, thereby suppressing the probability of the target model generating content on the same theme again. Understandably, the target user's emotional expressions such as laughter, sighs, anger, concise answers, or low-density information can also be used as current question-and-answer feedback information.

[0108] In an optional embodiment of the present invention, storing the current interaction request data, the current interaction response data, the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information in the local knowledge base may include: performing text vectorization processing on the current interaction request data and the current interaction response data to obtain vectorized text data of the current interaction request data and vectorized text data of the current interaction response data; performing feedback encoding on the current question-and-answer feedback information to obtain the encoding vector of the current question-and-answer feedback information; and storing the vectorized text data of the current interaction request data, the vectorized text data of the current interaction response data, the encoding vector of the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information in the local knowledge base.

[0109] The vectorized text data can be obtained by vectorizing the current interaction request data or the current interaction response data. The encoded vector of the current question-and-answer feedback information can be a low-dimensional dense numerical vector obtained by standardizing the vector encoding of the current question-and-answer feedback information.

[0110] In this embodiment of the invention, when storing the current interaction request data, current interaction response data, current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information into a local knowledge base, the current interaction request data and current interaction response data can first be converted into text format. Then, for the converted text-based request data and response data, a text vectorization algorithm can be used to process them separately, generating vectorized text data of the current interaction request data and vectorized text data of the current interaction response data. For the current question-and-answer feedback information with feedback attributes, a dedicated feedback encoding strategy is used for encoding to generate corresponding encoding vectors. Furthermore, the vectorized text data of the current interaction request data, the vectorized text data of the current interaction response data, the encoding vector of the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information can be associated and stored uniformly in the local knowledge base.

[0111] It should be noted that this solution employs a triple security mechanism in the data interaction phase to ensure the security and compliance of data transmission across private platforms. Firstly, when transmitting data across private platforms, it utilizes two mature short-range transmission technologies: Bluetooth Low Energy (BLE) and Wireless Fidelity (WiFi), ensuring stability and underlying security during transmission. Secondly, for core sensitive data such as text tokens and feature vectors, this solution employs a dedicated transmission mechanism for targeted protection, effectively preventing data leakage and ensuring the integrity and confidentiality of transmitted content. Thirdly, users have complete control over all data they generate, including full permissions to view, modify, delete, and authorize data. Furthermore, the overall data processing flow fully complies with current stringent data protection regulations, meeting compliance requirements.

[0112] This invention obtains the current interaction request data input by the target user and extracts multimodal features from the current interaction request data to obtain a first multimodal fusion feature vector of the current interaction request data. Further, multimodal features are extracted from each historical knowledge question-and-answer data in the local knowledge base to obtain a second multimodal fusion feature vector of each historical knowledge question-and-answer data. After obtaining the first and second multimodal fusion feature vectors, a retrieval recall score is calculated between the first and second multimodal fusion feature vectors. Based on the retrieval recall score, target historical knowledge question-and-answer data is selected from each historical knowledge question-and-answer data, thereby generating large-scale model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data. After obtaining the large-scale model interaction data, the target large-scale model is requested to generate text interaction response data for the current interaction request data based on the large-scale model interaction data. The text interaction response data is then converted into the current interaction response data of the current interaction request data using a text-to-speech model. This solution addresses the data leakage and privacy security risks existing in the data processing flow of existing RAG systems, ensuring user data security and thus improving the security and reliability of the human-computer interaction process.

[0113] To more clearly illustrate the technical solutions provided in the embodiments of the present invention, the embodiments of the present invention provide a reasoning question-and-answer method. Figure 5 This is a flowchart of a reasoning question-answering method provided in Embodiment 2 of the present invention. In a specific example, such as Figure 5 As shown, this reasoning question answering method is based on an edge-cloud collaborative reasoning question answering system, namely the PH-RAG system described in this application. This system consists of three functional modules: audio equipment, APP terminal, and cloud service. The three modules work together through a communication protocol to complete the reasoning question answering process.

[0114] Among them, the audio equipment serves as the terminal carrier for user interaction and undertakes three core functions: first, the input and output of voice interaction, realizing the collection of user voice commands and the playback of the final voice answer; second, the secure storage of personal privacy data, ensuring the integrity and privacy of users' personal preference history data; and third, the local PH recall calculation to improve response efficiency and reduce data transmission costs.

[0115] As a communication intermediary between audio equipment and cloud services, the APP terminal is responsible for data transmission adaptation and format conversion processing, specifically including: bidirectional data forwarding between audio equipment and cloud services, tokenization encoding of user interaction data, and detoxification decoding of cloud feedback data.

[0116] As the core computing node of the system, the cloud service integrates multiple neural network models and algorithm modules, and is responsible for completing complex computing tasks in the reasoning and question-answering process, including automatic speech recognition (ASR), text vectorization, audio feature encoding, large language model inference (LLM), user feedback information encoding, and text-to-speech (TTS).

[0117] Specifically, when a user initiates voice interaction through an audio device, the PH-RAG system can first preprocess the acquired user voice signal to calculate its frequency domain feature information, such as the Mel spectrum or MFCC cepstral coefficients. Subsequently, it can extract text features and audio features from the preprocessed user voice signal, and then fuse the two types of features to obtain a fused feature vector.

[0118] Furthermore, the audio equipment can access locally stored historical data of user preferences, perform calculations on the fused feature vector using the PH recall algorithm to generate a corresponding PH-Score, and then, based on this PH-Score, recall the top-k historical question-answer pairs that best match the current user's question from the historical data of user preferences, and tokenize them.

[0119] Furthermore, the app can detoxify the tokens of the top-k historical question-and-answer pairs, restoring them to a text format recognizable by the cloud-based large language model. It then structurally combines the current user question, the detoxified historical question-and-answer pairs, and preset prompts to generate a complete reasoning request text, which is then transmitted to the cloud service. The cloud service can invoke the large language model to perform reasoning calculations and generate the corresponding text answer. Further, the cloud service can invoke a TTS model with user voiceprint replication capabilities to convert the text answer into a speech signal matching the user's voiceprint features. This speech signal is transmitted from the app to the audio device, which then plays it back to the user, completing the entire reasoning question-and-answer process.

[0120] Figure 6 This is a flowchart of a data storage and archiving method provided in Embodiment 2 of the present invention. In a specific example, such as... Figure 6As shown, during the interaction with the PH-RAG system, the system needs to store and archive user questions, answers, and feedback. After the user completes voice input through the audio device, the APP terminal first preprocesses the acquired voice signal, converting the original voice time-domain information into Mel spectral features or MFCC frequency-domain features. Based on this, the cloud-based automatic speech recognition model simultaneously performs the core speech-to-text processing, generating not only the corresponding text content but also extracting text semantic features and audio semantic features. Furthermore, these text and audio semantic features are processed by the feature fusion module and stored in the local audio device's storage unit. Simultaneously, the cloud-based LLM model generates the text answer to the current question, which, after tokenization by the APP terminal, is also stored in the audio device. In addition, for the user's voice feedback, the PH-RAG system can obtain the user's personalized preference information for this question and answer through feedback encoding, which is ultimately stored in the audio device.

[0121] The reasoning-based question-answering method of this invention constructs an architecture that coordinates audio equipment, APP terminals, and cloud services. It achieves comprehensive information acquisition through the fusion of audio and text features, accurately recalls preferred historical question-answer pairs using the PH recall algorithm, and simultaneously stores core data on user-controllable audio equipment, forming a comprehensive optimized solution of "collaborative processing + accurate recall + local storage." This solution addresses the data leakage and privacy security risks inherent in existing RAG systems during data processing, ensuring user data security and thus improving the security and reliability of human-computer interaction.

[0122] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information (such as facial information, voice information, etc.) involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0124] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0125] Example 3

[0126] Figure 7 This is a schematic diagram of a human-computer interaction device provided in Embodiment 3 of the present invention, as shown below. Figure 7As shown, the device includes: a current interaction request data acquisition module 310, a target historical knowledge question-and-answer data determination module 320, a large model interaction data generation module 330, and a current interaction response data generation module 340, wherein:

[0127] The current interaction request data acquisition module 310 is used to acquire the current interaction request data input by the target user.

[0128] The target historical knowledge question and answer data determination module 320 is used to retrieve the target historical knowledge question and answer data corresponding to the current interaction request data input by the target user from the local knowledge base.

[0129] The large model interaction data generation module 330 is used to generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data.

[0130] The current interaction response data generation module 340 is used to generate current interaction response data of the current interaction request data based on the target large model of the large model interaction data request.

[0131] This invention acquires the current interaction request data input by the target user and retrieves the local knowledge base based on this data to obtain the target historical knowledge question-and-answer data corresponding to the current interaction request data. Further, it generates large-scale model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data. After obtaining the large-scale model interaction data, it requests the target large-scale model to generate the current interaction response data for the current interaction request data. This solution addresses the data leakage and privacy security risks present in existing RAG systems during data processing, ensuring user data security and thus improving the security and reliability of the human-computer interaction process.

[0132] Optionally, the target historical knowledge question-and-answer data determination module 320 is specifically used for: extracting multimodal features from the current interaction request data to obtain a first multimodal fusion feature vector of the current interaction request data; extracting multimodal features from each historical knowledge question-and-answer data in the local knowledge base to obtain a second multimodal fusion feature vector of each historical knowledge question-and-answer data; calculating a retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector; and filtering the target historical knowledge question-and-answer data from each historical knowledge question-and-answer data according to the retrieval recall score.

[0133] Optionally, if the current interaction request data is voice interaction request data, the target historical knowledge question-and-answer data determination module 320 is further configured to: convert the current interaction request data into text interaction request data; extract text features from the text interaction request data to obtain text semantic features of the current interaction request data; extract audio features from the current interaction request data to obtain audio semantic features of the current interaction request data; wherein the audio semantic features include at least one of emotion features, intent features, entity features, and speech features; and concatenate the text semantic features and the audio semantic features to obtain a first multimodal fusion feature vector of the current interaction request data.

[0134] Optionally, the target historical knowledge question and answer data determination module 320 is further configured to: use a preset word segmenter to perform word segmentation processing on the text interaction request data to obtain the word segmentation processing result of the text interaction request data; and use a text vector model to extract text features from the word segmentation processing result of the text interaction request data to obtain the text semantic features of the current interaction request data.

[0135] Optionally, the target historical knowledge question-and-answer data determination module 320 is further configured to: extract the frequency domain information of the current interaction request data; extract audio features from the frequency domain information of the current interaction request data to obtain the audio semantic features of the current interaction request data.

[0136] Optionally, the target historical knowledge question and answer data determination module 320 is further configured to: extract key information features and semantic information features from the question and answer interaction data in the historical knowledge question and answer data; extract feedback information features from the answer feedback data in the historical knowledge question and answer data; and fuse the key information features, the semantic information features, and the feedback information features to obtain a second multimodal fusion feature vector of the historical knowledge question and answer data.

[0137] Optionally, the target historical knowledge question-answering data determination module 320 is also used to: calculate the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector based on the following formula:

[0138] ;

[0139] ;

[0140] ;

[0141] in, The key vector matrix of the historical knowledge question-and-answer data. Let be the value vector matrix of the historical knowledge question and answer data. and Together they constitute the second multimodal fusion feature vector. The weight matrix applicable to the key vector matrix. The weight matrix to which the value vector matrix is ​​applied. These are the key information features of the question-and-answer interaction data in the historical knowledge question-and-answer data. The semantic information features of the question-and-answer interaction data in the historical knowledge question-and-answer data, The feedback information features of the answer feedback data in the historical knowledge question-and-answer data. For concatenation functions, The retrieval recall score is given. It is a feedforward neural network. For the attention calculation function, This is the first multimodal fusion feature vector of the current interactive request data.

[0142] Optionally, the large model interaction data generation module 330 is specifically used to: determine the large model interaction prompt template; and fill the large model interaction prompt template according to the target historical knowledge question and answer data and the current interaction request data to obtain the large model interaction data.

[0143] Optionally, the current interaction response data is voice-formatted interaction response data; the current interaction response data generation module 340 is specifically used to: generate text interaction response data of the current interaction request data based on the target large model interaction data request; and convert the text interaction response data into the current interaction response data of the current interaction request data through a text-to-speech model.

[0144] Optionally, the above-mentioned device may further include an information feedback module, used to obtain the current question-and-answer feedback information of the target user to the current interaction response data; calculate the interaction weight of the current question-and-answer feedback information based on the parsing result of the current question-and-answer feedback information; and store the current interaction request data, the current interaction response data, the current question-and-answer feedback information and the interaction weight of the current question-and-answer feedback information in the local knowledge base.

[0145] Optionally, the information feedback module is specifically used for: performing text vectorization processing on the current interaction request data and the current interaction response data to obtain vectorized text data of the current interaction request data and vectorized text data of the current interaction response data; performing feedback encoding on the current question-and-answer feedback information to obtain the encoding vector of the current question-and-answer feedback information; and storing the vectorized text data of the current interaction request data, the vectorized text data of the current interaction response data, the encoding vector of the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information in the local knowledge base.

[0146] The aforementioned human-computer interaction device can execute the human-computer interaction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the human-computer interaction method provided in any embodiment of the present invention.

[0147] Since the human-computer interaction device described above is a device capable of executing the human-computer interaction method in the embodiments of the present invention, those skilled in the art can understand the specific implementation methods and various variations of the human-computer interaction device in this embodiment based on the human-computer interaction method described in the embodiments of the present invention. Therefore, how the human-computer interaction device implements the human-computer interaction method in the embodiments of the present invention will not be described in detail here. Any device used by those skilled in the art to implement the human-computer interaction method in the embodiments of the present invention falls within the scope of protection of this application.

[0148] Example 4

[0149] Figure 8 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0150] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as human-computer interaction methods.

[0153] In some embodiments, the human-computer interaction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the human-computer interaction method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the human-computer interaction method by any other suitable means (e.g., by means of firmware).

[0154] Optionally, the human-computer interaction method may include: acquiring current interaction request data input by a target user; retrieving a local knowledge base based on the current interaction request data input by the target user to obtain target historical knowledge question-and-answer data corresponding to the current interaction request data; generating large model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data; and requesting a target large model to generate current interaction response data for the current interaction request data based on the large model interaction data.

[0155] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0157] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0159] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0160] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0161] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the human-computer interaction method provided in any embodiment of this application. This program product and the human-computer interaction methods disclosed in the embodiments of this application belong to the same inventive concept, and therefore will not be described in detail here.

[0162] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0163] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A human-computer interaction method, characterized in that, include: Obtain the current interaction request data input by the target user; The local knowledge base is retrieved based on the current interaction request data input by the target user to obtain the target historical knowledge question and answer data corresponding to the current interaction request data; Generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data; Based on the large model interaction data request target large model, the current interaction response data of the current interaction request data is generated.

2. The method according to claim 1, characterized in that, The step of retrieving the local knowledge base based on the current interaction request data input by the target user to obtain the target historical knowledge question and answer data corresponding to the current interaction request data includes: Multimodal features are extracted from the current interaction request data to obtain the first multimodal fusion feature vector of the current interaction request data; Multimodal features are extracted from each historical knowledge question and answer data in the local knowledge base to obtain a second multimodal fusion feature vector for each historical knowledge question and answer data. Calculate the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector; The target historical knowledge question and answer data is filtered from each of the historical knowledge question and answer data based on the retrieval recall score.

3. The method according to claim 2, characterized in that, If the current interaction request data is voice interaction request data, then the step of extracting multimodal features from the current interaction request data to obtain the first multimodal fusion feature vector of the current interaction request data includes: Convert the current interaction request data into text interaction request data; Text feature extraction is performed on the text interaction request data to obtain the text semantic features of the current interaction request data; Audio features are extracted from the current interaction request data to obtain the audio semantic features of the current interaction request data; wherein, the audio semantic features include at least one of emotion features, intent features, entity features, and speech features; The text semantic features and the audio semantic features are concatenated to obtain the first multimodal fusion feature vector of the current interaction request data.

4. The method according to claim 3, characterized in that, The step of extracting text features from the text interaction request data to obtain the text semantic features of the current interaction request data includes: The text interaction request data is segmented using a preset word segmenter to obtain the word segmentation result of the text interaction request data. The text feature extraction of the word segmentation result of the text interaction request data is performed using a text vector model to obtain the text semantic features of the current interaction request data.

5. The method according to claim 3, characterized in that, The step of extracting audio features from the current interaction request data to obtain the audio semantic features of the current interaction request data includes: Extract the frequency domain information of the current interactive request data; Audio features are extracted from the frequency domain information of the current interactive request data to obtain the audio semantic features of the current interactive request data.

6. The method according to claim 2, characterized in that, The step of extracting multimodal features from each historical knowledge question-and-answer data in the local knowledge base to obtain a second multimodal fusion feature vector for each historical knowledge question-and-answer data includes: Extract key information features and semantic information features from the question-and-answer interaction data in the historical knowledge question-and-answer data; Extract feedback information features from the answer feedback data in the historical knowledge question and answer data; By fusing the key information features, the semantic information features, and the feedback information features, a second multimodal fusion feature vector of the historical knowledge question-and-answer data is obtained.

7. The method according to claim 6, characterized in that, The calculation of the retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector includes: The retrieval recall score between the first multimodal fusion feature vector and the second multimodal fusion feature vector is calculated based on the following formula: ； ； ； in, The key vector matrix of the historical knowledge question-and-answer data. The value vector matrix of the historical knowledge question-and-answer data. and Together they constitute the second multimodal fusion feature vector. The weight matrix applicable to the key vector matrix. The weight matrix to which the value vector matrix is ​​applied. These are the key information features of the question-and-answer interaction data in the historical knowledge question-and-answer data. The semantic information features of the question-and-answer interaction data in the historical knowledge question-and-answer data, The feedback information features of the answer feedback data in the historical knowledge question-and-answer data. For concatenation functions, The retrieval recall score is given. It is a feedforward neural network. For the attention calculation function, This is the first multimodal fusion feature vector of the current interactive request data.

8. The method according to any one of claims 1 to 7, characterized in that, The step of generating large model interaction data based on the target historical knowledge question-and-answer data and the current interaction request data includes: Determine the template for interactive prompts for the large model; The large model interaction prompt template is populated with the target historical knowledge question and answer data and the current interaction request data to obtain the large model interaction data.

9. The method according to any one of claims 1 to 7, characterized in that, The current interactive response data is interactive response data in voice format; The step of generating the current interaction response data of the current interaction request data based on the target large model interaction data request includes: Based on the large model interaction data request target large model, generate text interaction response data for the current interaction request data; The text-to-speech model is used to convert the text interaction response data into the current interaction response data of the current interaction request data.

10. The method according to any one of claims 1 to 7, characterized in that, After generating the current interaction response data of the current interaction request data based on the target large model according to the large model interaction data request, the method further includes: Obtain the current question-and-answer feedback information of the target user in response to the current interaction response data; Calculate the interaction weight of the current question and answer feedback information based on the parsing result of the current question and answer feedback information; The current interaction request data, the current interaction response data, the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information are stored in the local knowledge base.

11. The method according to claim 10, characterized in that, The step of storing the current interaction request data, the current interaction response data, the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information into the local knowledge base includes: The current interaction request data and the current interaction response data are processed into text vectorization to obtain vectorized text data of the current interaction request data and vectorized text data of the current interaction response data; The current question-and-answer feedback information is encoded to obtain the encoding vector of the current question-and-answer feedback information; The vectorized text data of the current interaction request data, the vectorized text data of the current interaction response data, the encoded vector of the current question-and-answer feedback information, and the interaction weight of the current question-and-answer feedback information are stored in the local knowledge base.

12. A human-computer interaction device, characterized in that, include: The current interaction request data acquisition module is used to acquire the current interaction request data input by the target user; The target historical knowledge question and answer data determination module is used to retrieve the target historical knowledge question and answer data corresponding to the current interaction request data input by the target user from the local knowledge base. The large model interaction data generation module is used to generate large model interaction data based on the target historical knowledge question and answer data and the current interaction request data. The current interaction response data generation module is used to generate current interaction response data of the current interaction request data based on the target large model interaction data request of the large model interaction data request.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the human-computer interaction method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the human-computer interaction method according to any one of claims 1-11.

15. A computer program product comprising a computer program, wherein, When the computer program is executed by the processor, it implements the human-computer interaction method according to any one of claims 1-11.

Images