Multi-modal large model dialogue method and system based on context awareness and medium

Abstract

The invention discloses a multi-modal large model dialogue method and system based on context awareness and a medium. Relates to the technical field of intelligent dialogues. According to the scheme, the dialogue request information and the perception feature information are generated in the context perception dialogue management model, and the answer strategy is determined according to the dialogue request information and the perception feature information; on the basis of various data of the user, system hot word auxiliary information provides personalized reply for the user and is more suitable for life links, accurate context perception is achieved, and enterprise product operation and development are served.

Description

Technical Field

[0001] The present invention relates to the field of intelligent dialogue technology, and in particular to a context-aware multimodal large-model dialogue method, system and medium. Background Art

[0002] The emergence of large models provides more intelligent capabilities for human-computer dialogue, and more and more industry applications have emerged to provide users with generative question-answering services. In the process of providing question-answering services, the existing technology mainly combines the user's current and historical dialogues, combines the knowledge base data to perform vector recall knowledge to answer user questions, or based on the user's current question-answering conclusion and the system's historical question-answering results probability calculation, outputs the most probable answer to the user.

[0003] The main issues are:

[0004] 1. In terms of user context perception, it fails to fully integrate the current system (environment, society, etc.) hot words, customer question emotional intentions, and system recommended content features. On the one hand, the accuracy of identifying user needs in context recognition is weak, and it fails to fully analyze whether the user's historical conversation is related to the current question, and answer it in combination with historical answers and user active feedback; on the other hand, it fails to provide support for the current product recommendations of the business system, thereby realizing the marketing and promotion of corporate products and generating more economic value.

[0005] 2. In terms of dialogue management, the current technical solution mainly records and archives user dialogues, and has not optimized the overall question-answering process. First, all questions in user dialogues need to go through the model generation module. For known questions, they are not directly answered in combination with non-model applications such as FAQ, which increases the resource consumption of the system and adds inaccuracy to the answers to already clear questions; secondly, for the dialogues that the system has responded to, the model optimization strategy is not generated in combination with user intentions and actual feedback, and the corpus dialogue data optimized for large and small models has not been formed to continuously update the accuracy of the system. Summary of the invention

[0006] In order to solve the technical problems existing in the existing dialogue systems, the purpose of the present invention is to provide a multimodal large-model dialogue method, system and medium based on context awareness, generate dialogue request information and perception feature information based on the context-aware dialogue management model, and determine the answer strategy according to the dialogue request information and perception feature information; based on various user data, the system hot word auxiliary information provides users with personalized replies while being more relevant to life links, achieving accurate context perception, and serving the company's product management and development.

[0007] The present invention is achieved through the following technical solutions:

[0008] This solution provides a context-aware multimodal large-model dialogue method, including:

[0009] Collecting a first conversation message and preprocessing the first conversation message;

[0010] Inputting the preprocessed first dialogue message into the context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: sentiment vector information, analysis information based on historical dialogue information and user portrait context intent, and system hot word auxiliary information;

[0011] Determine the answer strategy based on the dialogue request information and the perception feature information;

[0012] A second dialog message is generated according to the answer strategy.

[0013] A further optimization scheme is that the preprocessing includes: extracting conversation features based on high-dimensional vector representation from the multimodal first conversation message, and the multimodal first conversation message includes one or more of text data, voice data, image data and video data.

[0014] A further optimization scheme is that the method for generating the emotion vector information includes:

[0015] Based on the attention mechanism and weighted fusion strategy, the sentiment features of the multimodal first conversation messages are integrated into a unified sentiment vector information representation:

[0016] E = Concat(E Speech , E Text , E Image )·W+b

[0017] Where E represents the sentiment vector; E Speech , E Text , E Image They represent speech features, text features and image features respectively, W is the weight matrix, b is the bias term; Concat() represents the connection function.

[0018] A further optimization scheme is that the method for generating analysis information based on historical conversation information and user portrait contextual intent includes:

[0019] Obtain the user's historical conversations and construct a record table of each historical conversation; the record table includes: time, question, system answer, question vector, system answer vector and user feedback;

[0020] The historical conversation feature index is composed of questions, answers and user feedback to calculate the historical question-answer similarity Si and the feedback index F. 踩 Like feedback index F 赞 :

[0021] Si=similarity(Hi,Hn)(Si∈[0,1])

[0022] Wherein, similarity() represents a similarity calculation function; Hi represents the historical conversation message of the i-th round; Hn represents the current conversation message;

[0023]

[0024] Among them, N represents the total number of preset historical dialogue messages, N 赞 Indicates the number of user like feedback rounds in the preset historical conversation message, N 踩 Indicates the number of rounds of user feedback in the preset historical dialogue message; N 空 Indicates the number of rounds in which the user has no feedback in the preset historical conversation messages.

[0025] A further optimization scheme is that the method for generating auxiliary information of hot words in the system includes:

[0026] According to the sentiment vector information, the downvote feedback index Fdownvote and the upvote feedback index Fupraise, it is determined whether there is a hot word matching the first conversation message in the hot word product library. If so, the matching hot word is used as the system hot word auxiliary information. Otherwise, the system hot word auxiliary information is not generated.

[0027] The hot word product library stores social conversation hot words and product feature hot words.

[0028] A further optimization scheme is that the answer strategy is determined according to the dialogue request information and the perception feature information, including the method:

[0029] S1, based on the dialogue request information and the perception feature information, the emotional state is judged. When the emotional state is not satisfactory, the guiding words are output as the reply; otherwise, the process goes to step S2;

[0030] S2, determine whether the current conversation request information is a real-time message, if so, proceed to step S3, otherwise, add the current conversation request information to the message queue, arrange the message queue according to the preset priority order, and proceed to step S3 according to the order of the message queue;

[0031] S3, calculate the semantic similarity between the current conversation request information and the historical conversation request information. When the semantic similarity is greater than the similarity threshold, call the FAQ question-answering interface as the answer object; when the semantic similarity is less than the similarity threshold, call the large model as the answer object;

[0032] The FAQ module replies to the first dialogue message based on the known knowledge base, and the large model is generated based on the known knowledge base combined with retrieval enhancement to provide an intelligent reply to the first dialogue message in the form of prompt word engineering application.

[0033] Distinguishing between known questions and answers still requires reasoning about question and answer data, improving the standardization of responses while reducing the resource consumption of the entire dialogue system; during the dialogue process, the conclusions of the context perception module and the dialogue strategy management module are combined with the user's emotional state to guide manual processing or friendly reminders to improve service satisfaction. On the other hand, combined with the queue consumption algorithm of non-real-time messages, the pressure indicators of the entire dialogue system are dynamically managed in the form of hierarchical priority queues, providing more marketing models for the commercial services of the product. Users can choose to recharge, VIP, etc. to improve the speed and efficiency of question and answer; after the dialogue is over, the current round of dialogue will be fed back to form a system dialogue expectation library, and the optimization module will optimize the model or adjust other module parameters to improve the accuracy of the entire question and answer and user perception.

[0034] A further optimization scheme is that the method for determining the emotional state includes:

[0035] Calculate the weighted average WAS of the similarity of the most recent N rounds of historical conversations:

[0036]

[0037] Where, d i represents the weight of the i-th round of historical dialogue, with a value range of [0,1]; Q represents the question of the current dialogue, Qi represents the question of the i-th round of historical dialogue, similarity() represents the similarity calculation function, Mi represents the similarity between the current dialogue question and the i-th round of historical dialogue; N represents the total number of historical dialogue rounds;

[0038] Click status value DLStatus: The value range of DLStatus is [0, 1]; Z represents the total number of preset second historical dialogue message rounds; Z i Indicates the second historical conversation message of round i;

[0039] Emotional state discrimination parameter JE=WAS*DLStatus*E.

[0040] A further optimization scheme is that in step S2, the message queue is arranged in a preset priority order, including the method:

[0041] In the message queue, three priorities are set by default. The consumption order of the same priority is determined by the arrival time. Messages of different priorities are determined by the following formula:

[0042] Priority queue pressure coefficient after this round of consumption

[0043] Among them, Wi is the number of waiting rounds for the current priority; Mi represents the remaining number of messages; Mi≠0.

[0044] This solution also provides a context-aware multimodal large model dialogue system, which is used to implement the context-aware multimodal large model dialogue method described in the above solution; the system includes:

[0045] A multimodal input processing module, used for collecting a first dialogue message and preprocessing the first dialogue message;

[0046] A context processing and dialogue management module, used to input the preprocessed first dialogue message into a context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: sentiment vector information, analysis information based on historical dialogue information and user portrait context intent, and system hot word auxiliary information;

[0047] A dialogue routing module is used to determine the answer strategy based on the dialogue request information and the perception feature information;

[0048] A generating module is used to generate a second dialogue message according to the answering strategy.

[0049] A further optimization scheme is that the context processing and dialogue management module includes:

[0050] Conversation memory, used to store user historical conversation data;

[0051] Hot word product library, used to store social conversation hot words and product feature hot words

[0052] The sentiment analysis unit is used to analyze user questions in combination with user historical conversation data and generate conversation request information and perception feature information.

[0053] The present solution also provides a computer-readable medium having a computer program stored thereon, and the computer program is executed by a processor to implement the context-aware multi-modal large model dialogue method as described in the above solution.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] 1. The present invention provides a context-aware multimodal large model dialogue method, system and medium; combines historical dialogue information and user portraits to supplement contextual intent and content; when answering through large language model reasoning, based on user portraits, combined with the company's business, product promotion indicators, current popular hot words and other external elements, to provide users with personalized answers while being more relevant to life links, achieving accurate context perception and serving the company's product management and development.

[0056] 2. The present invention provides a context-aware multimodal large-model dialogue method, system and medium. Before the user dialogue reaches the generation module, the dialogue routing model is used to quickly distinguish whether the question and answer are known or need to be inferred, thereby improving the standardization of responses and reducing the resource consumption of the entire dialogue system. During the dialogue, the context-aware module and the dialogue strategy management module are combined to judge the user's emotional state, guide manual processing or friendly reminders, and improve service satisfaction. After the dialogue ends, the current round of dialogue is fed back to form a system dialogue expectation library, and the optimization module optimizes the model or adjusts other module parameters to improve the accuracy of the entire system's question and answer rate and user perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0058] Figure 1 This is a flowchart of a multimodal large model dialogue method based on context awareness;

[0059] Figure 2 This is a schematic diagram of the structure of a multimodal large-model dialogue system based on context awareness;

[0060] Figure 3 This is a diagram of the Q&A visualization interactive page;

[0061] Figure 4 This is a diagram of the dialogue memory bank;

[0062] Figure 5 This is a business process diagram of the context processing and dialogue management module;

[0063] Figure 6 Schematic diagram of dialogue request information and perception feature information case;

[0064] Figure 7 Determine the schematic diagram of the principle for answering strategy;

[0065] Figure 8 Determine a flowchart for the answer strategy;

[0066] Fig. 9 This is a schematic diagram of an example of message queue consumption priority;

[0067] Fig.10 This is a schematic diagram of the second example of message queue consumption priority;

[0068] Fig.11Schematic diagram of the workflow for optimizing intelligent question-answering dialogue data. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0070] Existing conversation methods and systems have (1) problems with improving the accuracy of context perception and accurately pushing customer service product marketing; the user side combines the user's context, historical conversation data, user portraits, etc., and the product side combines indicators such as corporate marketing business product characteristics to jointly generate speech to answer user questions while promoting corporate products. (2) Conversation management issues, one is the routing distinction after the question arrives, specifically whether it can go directly to FAQ or needs to go through a large language model problem; the second is user conversation queues, real-time and non-real-time conversation management, user emotional status update conversation management, etc. For example, during a conversation, if the user frequently angrily clicks, then the conversation needs to be guided manually or given friendly reminders; the third is the management of feedback conversation data after the conversation is completed to form a conversation corpus, whether it is positive feedback or negative feedback data. In view of this, this solution provides the following embodiments to solve the above technical problems:

[0071] Example 1

[0072] This embodiment provides a multi-modal large model dialogue method based on context awareness, such as Figure 1 As shown, including:

[0073] Step 1: Collect a first conversation message and preprocess the first conversation message; the preprocessing includes: extracting conversation features based on high-dimensional vector representation from the multimodal first conversation message, wherein the multimodal first conversation message includes one or more of text data, voice data, image data and video data.

[0074] Step 2: Input the preprocessed first dialogue message into the context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: sentiment vector information, analysis information based on historical dialogue information and user portrait context intent, and system hot word auxiliary information;

[0075] The method for generating the emotion vector information comprises:

[0076] Based on the attention mechanism and weighted fusion strategy, the sentiment features of the multimodal first conversation messages are integrated into a unified sentiment vector information representation:

[0077] E = Concat(E Speech , E Text , EImage )·W+b

[0078] Where E represents the sentiment vector; E Speech , E Text , E Image They represent speech features, text features and image features respectively, W is the weight matrix, b is the bias term; Concat() represents the connection function.

[0079] The method for generating analysis information based on historical conversation information and user portrait contextual intent includes:

[0080] Obtain the user's historical conversations and construct a record table of each historical conversation; the record table includes: time, question, system answer, question vector, system answer vector and user feedback;

[0081] The historical conversation feature index is composed of questions, answers and user feedback to calculate the historical question-answer similarity Si and the feedback index F. 踩 Like feedback index F 赞 :

[0082] Si=similarity(Hi,Hn)(Si∈[0,1])

[0083] Wherein, similarity() represents a similarity calculation function; Hi represents the historical conversation message of the i-th round; Hn represents the current conversation message;

[0084]

[0085] Among them, N represents the total number of preset historical dialogue messages, N 赞 Indicates the number of user like feedback rounds in the preset historical conversation message, N 踩 Indicates the number of rounds of user feedback in the preset historical dialogue message; N 空 Indicates the number of rounds in which the user has no feedback in the preset historical conversation messages.

[0086] The method for generating system hot word auxiliary information includes:

[0087] According to the sentiment vector information, the downvote feedback index Fdownvote and the upvote feedback index Fupraise, it is determined whether there is a hot word matching the first conversation message in the hot word product library. If so, the matching hot word is used as the system hot word auxiliary information. Otherwise, the system hot word auxiliary information is not generated.

[0088] The hot word product library stores social conversation hot words and product feature hot words.

[0089] Step 3: Determine the answer strategy based on the dialogue request information and the perception feature information; this step includes the following methods:

[0090] S1, based on the dialogue request information and the perception feature information, the emotional state is judged. When the emotional state is not satisfactory, the guiding words are output as the reply; otherwise, the process goes to step S2;

[0091] S2, determine whether the current conversation request information is a real-time message, if so, proceed to step S3, otherwise, add the current conversation request information to the message queue, arrange the message queue according to the preset priority order, and proceed to step S3 according to the order of the message queue;

[0092] S3, calculate the semantic similarity between the current conversation request information and the historical conversation request information. When the semantic similarity is greater than the similarity threshold, call the FAQ question-answering interface as the answer object; when the semantic similarity is less than the similarity threshold, call the large model as the answer object;

[0093] The FAQ module replies to the first dialogue message based on the known knowledge base, and the large model is generated based on the known knowledge base combined with retrieval enhancement to provide an intelligent reply to the first dialogue message in the form of prompt word engineering application.

[0094] The method for determining the emotional state comprises:

[0095] Calculate the weighted average WAS of the similarity of the most recent N rounds of historical conversations:

[0096]

[0097] Where, d i represents the weight of the i-th round of historical dialogue, with a value range of [0,1]; Q represents the question of the current dialogue, Qi represents the question of the i-th round of historical dialogue, similarity() represents the similarity calculation function, Mi represents the similarity between the current dialogue question and the i-th round of historical dialogue; N represents the total number of historical dialogue rounds;

[0098] Click status value DLStatus: The value range of DLStatus is [0, 1]; Z represents the total number of preset second historical dialogue message rounds; Z i Indicates the second historical conversation message of round i;

[0099] Emotional state discrimination parameter JE=WAS*DLStatus*E.

[0100] An emotional state threshold is set, and the emotional state discrimination parameter is compared with the emotional state threshold to judge the emotional state of the user. If the emotional state discrimination parameter is within a certain range, the user's emotional state is good, otherwise the user's emotional state is poor.

[0101] Step 4: Generate a second dialogue message according to the answer strategy.

[0102] Example 2

[0103] This embodiment provides a context-aware multimodal large model dialogue system, which is used to implement the context-aware multimodal large model dialogue method described in Example 1; Figure 2 As shown, the system comprises:

[0104] A multimodal input processing module, used for collecting a first dialogue message and preprocessing the first dialogue message;

[0105] A context processing and dialogue management module, used to input the preprocessed first dialogue message into a context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: sentiment vector information, analysis information based on historical dialogue information and user portrait context intent, and system hot word auxiliary information;

[0106] Specifically, it also includes a visual interaction module, which is used to collect the first conversation message, and mainly provides users with a multi-modal question and answer interactive interface, including an input box, an output content display page, and functional modules such as likes and dislikes; the whole is divided into four parts, including a user input area, a question and answer display area, a like and dislike area, and an area for uploading local documents. Users can long press "enter" to input voice, or use the upload button to provide existing recording files, pictures, and text files for question and answer. The visual interaction module mainly provides users with the entry capability of multi-modal interaction. The question and answer visual interaction page is as follows: Figure 3 shown.

[0107] The multimodal input processing module is mainly divided into three submodules, namely, speech processing submodule, image processing submodule, and text preprocessing submodule;

[0108] Modern users' communication is no longer limited to text, but also includes multi-modal signals such as voice, image, and video. By introducing multi-modal input processing capabilities, the voice, image, document and other information input by users are processed and integrated with text input for analysis to generate a more complete and accurate response. Among them, speech feature extraction is mainly based on the structure of combining CNN and RNN to extract speech signal features, and at the same time, combined with ASR to convert speech into text for use by subsequent modules. CNN is used to capture the frequency and time domain features of speech, and RNN is used to learn the time series dependency of speech signals. The trained convolutional layer extracts important speech features such as Mel spectrograms, and then semantic modeling is performed through the bidirectional LSTM layer to finally form a speech feature representation vector. Text feature extraction is mainly encoded through a pre-trained transformer model to obtain context-related semantic feature representations. Models such as BERT and T5 extract deep semantic information through a multi-layer self-attention mechanism and convert text sequences into high-dimensional vector representations for multi-modal fusion. Image feature extraction mainly uses pre-trained convolutional neural networks (such as ResNet and EfficientNet) to extract key image features. The visual features of key frames are formed into high-dimensional vectors after global average pooling and feature normalization to provide input feature representation. For video feature extraction, a three-dimensional convolutional neural network (3D-CNN) is used to capture dynamic information in the time dimension.

[0109] The context processing and dialogue management module includes:

[0110] Conversation memory, used to store user historical conversation data;

[0111] Hot word product library, used to store social conversation hot words and product feature hot words

[0112] The sentiment analysis unit is used to analyze user questions in combination with user historical conversation data and generate conversation request information and perception feature information.

[0113] The dialogue memory is as follows: Figure 4 As shown in the figure, it mainly stores two types of data, one is user-side data, and the other is system-side data. For historical conversations, the main storage includes user ID, time, question, answer-system, vector-question, vector-system answer, user feedback, etc., see Appendix 1 for examples; user ID is used as the primary key to distinguish data attribution, and user portrait data examples are shown in Appendix 2; fields include user ID, age, gender, characteristics, consumption ability, target product, catchphrase, number of conversations in the past 30 days, number of likes in the past 30 days, number of dislikes in the past 30 days, etc. For system-side data, it mainly stores the product and policy data currently configured by the system, as well as the hot words commonly used in the current social environment, so as to generate answers that are closer to user and system needs when answering user questions later.

[0114] Appendix 1: Conversation Record

[0115]

[0116] Appendix 2 User Characteristics Record Table

[0117]

[0118] The business process of the context processing and dialogue management module is shown in the attached figure. Figure 5 As shown in the figure, it includes global features and multi-level information extraction, combined with an enhanced dialogue memory library. After analyzing the user's questions, it generates the final dialogue request information and perception feature information for the next module to use.

[0119] Specific examples are attached Figure 6 , starting with user questions, the context recognition module's reference recognition, intent recognition, emotion recognition and other sub-modules and the business system library enhance the user questions and hand them over to the dialogue generation module to answer the questions. The system pre-sets responses and question-answering processes for different dialogue scenarios, such as responses to emotional breakdown and network timeout. Based on the emotion vector E, the system generates responses with different tones. For example, in an angry state, the system will avoid using offensive language, while in a sad state, the system will provide comfort and supportive statements.

[0120] The method for generating the analysis information of historical conversation information and user portrait contextual intent includes: the time range of historical conversation is configured by the system, which can be all current conversations or conversations in the recent period, and the historical rounds and the similarity of the current questions are referred to. The closer the questions are and the more similar the questions are, the greater the weight is, and the thumbs-up and thumbs-down feedback indicators are composed of thumbs-down feedback indicators. Based on the emotion vector and thumbs-up and thumbs-down feedback indicators, decide whether to recommend the current product library to the user; combined with the user ID, according to the user portrait characteristics, pass the user's favorite hot words to the next module.

[0121] The data sample of the request conversation routing interface is as follows:

[0122]

[0123] The hot word product library mainly stores hot words commonly used in the current system social dialogue, such as "the first cup of milk tea in autumn"; as well as key information such as the introduction and features of the company's main products at the current stage. All data is presented as memory data when applied to improve access efficiency. While answering user questions through the hot word product library to assist the big model, it also combines user characteristics and enterprise product features to potentially push relevant information, better helping enterprises to provide customer service and product marketing promotion.

[0124] A dialogue routing module is used to determine the answer strategy based on the dialogue request information and the perception feature information;

[0125] The answer strategy determination process is as follows Figure 7-8 As shown in the figure, first, after the user's current round of questions arrives, the emotions and likes of the historical conversation may not be good, and the customer needs to be guided to manual service to reduce the breeding of dissatisfaction. For example, the system determines that the current user has asked similar questions several times in a row, and the emotional probability is already angry, and returns the default answer: "Dear, we have received your question and generated a work order. The work order processing result will be fed back to you as soon as possible."; Secondly, the real-time and non-real-time messages are diverted to improve the system response efficiency; Thirdly, there may be system standard answers to existing questions, so it is only necessary to match the FAQ questions with the answers. For questions that require reasoning answers, the big model can be used for reasoning. After the user dialogue, the emotional state and likes status are recorded to provide a basis for whether the next round of dialogue will adopt the big model reasoning answer or the system default answer. At the same time, the recorded dialogue log provides data basis for subsequent model optimization.

[0126] For non-real-time messages, three priorities are set by default in the message queue. The consumption order of the same priority is determined by the arrival time. Messages of different priorities are determined by the following formula. The main parameters are the number of waiting rounds for the priority (Wi), the remaining number of messages (Mi), and the queue pressure coefficient (QP). During actual consumption, the i with the maximum QP is the priority queue to be consumed next time. If the remaining number of messages Mi is equal to 0, the priority judgment is skipped directly. The specific formula is as follows:

[0127] Priority queue pressure coefficient after this round of consumption Where i is the priority, Mi≠0.

[0128] Detailed consumption examples are shown in the attached Fig. 9 and attached Fig.10 shown.

[0129] The dialogue routing module is a separate module that mainly judges based on user questions and context processing and the recognition results of the dialogue management module, integrates existing knowledge question and answer content, and supports fast replies without the need for large model reasoning;

[0130] The FAQ module mainly contains frequently used question and answer pairs. This module mainly associates questions based on their similarity and directly answers existing knowledge, for example: "Q: How to apply for the 169 package? A: There are three main ways to apply for the 169 package. The first is...". The main sources of knowledge are two aspects. One is system import. Combined with the previously known question and answer data, the question and answer pairs are imported by default. The other is that with the continuous optimization of the question and answer system, the FAQ knowledge base is updated for questions and answers that users are not satisfied with or are manually accessed, so as to improve the knowledge coverage of the module.

[0131] The large model module is mainly responsible for answering questions about unknown knowledge. It combines the knowledge base for search enhancement and generation, provides intelligent question-answering services in the form of prompt word engineering applications, and increases the scalability of question-answering generation. For example, when a user asks a question that does not exist in the current FAQ, but the conversation still requires a system response rather than a manual response, the question is matched with the knowledge vector in the knowledge base, and as many relevant original texts as possible are found and sorted before being sent to the large language model for reasoning question-answering.

[0132] A generating module is used to generate a second dialogue message according to the answering strategy.

[0133] It also includes a vector knowledge base and optimization module, which mainly stores large language model knowledge and generates text vectorization results, while providing dialogue data support for dialogue strategy updates.

[0134] The vector knowledge base and optimization module mainly stores the knowledge of the large language model and generates the vectorized results of the text, and also provides dialogue data support for dialogue strategy updates. The former cooperates with the large language model to provide support for improving the accuracy of question answering, such as Fig.11 As shown in the figure, the intelligent question-answering dialogue corpus optimization workflow mainly consists of the following three steps:

[0135] a. Sample and analyze historical user questions and answers, combine the semantic analysis of user questions with a small natural language processing model, and update the system's hot word library.

[0136] b. Sample and analyze the conversation records of manual customer service answers, and have manual experts evaluate the question-and-answer process to confirm whether the conversation can be included in the FAQ and question-and-answer knowledge base, providing a high-quality data set for subsequent training and fine-tuning of large and small models.

[0137] c. Sample analysis of the question and answer pairs clicked by users, and after evaluation by manual experts, confirm whether the questions have been responded to and answered. The knowledge base update process is consistent with the process of sampling and analyzing manual customer service questions and answers, and a targeted customer return visit process is added. The return visit method can be a system message or a pop-up message push, etc.

[0138] Example 3

[0139] This embodiment provides a computer-readable medium on which a computer program is stored. The computer program is executed by a processor to implement the context-aware multimodal large model dialogue method as described in Embodiment 1; specifically, the following steps are implemented:

[0140] Step 1: collecting a first conversation message and preprocessing the first conversation message;

[0141] Step 2: Input the preprocessed first dialogue message into the context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: sentiment vector information, analysis information based on historical dialogue information and user portrait context intent, and system hot word auxiliary information;

[0142] Step 3: Determine the answer strategy based on the dialogue request information and the perception feature information;

[0143] Step 4: Generate a second dialogue message according to the answer strategy.

[0144] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A context-aware multimodal large-model dialogue method, characterized in that: include: Collecting a first conversation message and preprocessing the first conversation message; Inputting the preprocessed first dialogue message into a context-aware dialogue management model to generate dialogue request information and perception feature information; The perceptual feature information includes: sentiment vector information, analysis information based on historical conversation information and user portrait contextual intent, and system hot word auxiliary information; Determine the answer strategy based on the dialogue request information and the perception feature information; A second dialog message is generated according to the answer strategy.

2. The context-aware multimodal large model dialogue method according to claim 1, characterized in that: The preprocessing includes: extracting a conversation feature represented by a high-dimensional vector from a multimodal first conversation message, wherein the multimodal first conversation message includes one or more of text data, voice data, image data, and video data.

3. The context-aware multimodal large model dialogue method according to claim 2 is characterized in that: The method for generating the emotion vector information comprises: Based on the attention mechanism and weighted fusion strategy, the sentiment features of the multimodal first conversation messages are integrated into a unified sentiment vector information representation: E=Concat(E Speech ,E Text ,E Image )·W+b Where E represents the sentiment vector; E Speech , E Text , E Image They represent speech features, text features and image features respectively, W is the weight matrix, b is the bias term; Concat() represents the connection function.

4. The context-aware multimodal large model dialogue method according to claim 3 is characterized in that: The method for generating analysis information based on historical conversation information and user portrait contextual intent includes: Obtain the user's historical conversations and construct a record table of each historical conversation; the record table includes: time, question, system answer, question vector, system answer vector and user feedback; The historical conversation feature index is composed of questions, answers and user feedback to calculate the historical question-answer similarity Si and the feedback index F. 踩 Like feedback index F 赞 : Si=similarity(Hi,Hn)(Si∈[0,1]) Wherein, similarity() represents a similarity calculation function; Hi represents the historical conversation message of the i-th round; Hn represents the current conversation message; Where N represents the total number of rounds of the preset first historical dialogue message, N 赞 Indicates the number of user like feedback rounds in the preset first historical conversation message, N 踩 Indicates the number of rounds of user feedback in the preset first historical dialogue message; N 空 Indicates the number of rounds in which the user has no feedback in the preset first historical dialogue message.

5. The context-aware multimodal large model dialogue method according to claim 4, characterized in that: The method for generating system hot word auxiliary information includes: According to the sentiment vector information, the downvote feedback index Fdownvote and the upvote feedback index Fupraise, it is determined whether there is a hot word matching the first conversation message in the hot word product library. If so, the matching hot word is used as the system hot word auxiliary information. Otherwise, the system hot word auxiliary information is not generated. The hot word product library stores social conversation hot words and product feature hot words.

6. The context-aware multimodal large model dialogue method according to claim 5, characterized in that: The method of determining the answer strategy according to the dialogue request information and the perception feature information includes: S1, based on the dialogue request information and the perception feature information, the emotional state is judged. When the emotional state is not satisfactory, the guiding words are output as the reply; Otherwise, proceed to step S2; S2, determine whether the current conversation request information is a real-time message, if so, proceed to step S3, otherwise, add the current conversation request information to the message queue, arrange the message queue according to the preset priority order, and proceed to step S3 according to the order of the message queue; S3, calculate the semantic similarity between the current conversation request information and the historical conversation request information. When the semantic similarity is greater than the similarity threshold, call the FAQ question-answering interface as the answer object; when the semantic similarity is less than the similarity threshold, call the large model as the answer object; The FAQ module replies to the first dialogue message based on the known knowledge base, and the large model is generated based on the known knowledge base combined with retrieval enhancement to provide an intelligent reply to the first dialogue message in the form of prompt word engineering application.

7. The context-aware multimodal large model dialogue method according to claim 6, characterized in that: The method for determining the emotional state comprises: Calculate the weighted average WAS of the similarity of the most recent N rounds of historical conversations: Where, d i represents the weight of the i-th round of historical dialogue, with a value range of [0,1]; Q represents the question of the current dialogue, Qi represents the question of the i-th round of historical dialogue, similarity() represents the similarity calculation function, Mi represents the similarity between the current dialogue question and the i-th round of historical dialogue; N represents the total number of historical dialogue rounds; Click status value DLStatus: The value range of DLStatus is [0, 1]; Z represents the total number of preset second historical dialogue message rounds; Z i Indicates the second historical conversation message of round i; Emotional state discrimination parameter JE=WAS*DLStatus*E.

8. The context-aware multimodal large model dialogue method according to claim 7, characterized in that: In step S2, the message queue is arranged in a preset priority order, including a method: In the message queue, three priorities are set by default. The consumption order of the same priority is determined by the arrival time. Messages of different priorities are determined by the following formula: Priority queue pressure coefficient after this round of consumption Among them, Wi is the number of waiting rounds for the current priority; Mi represents the remaining number of messages; Mi≠0.

9. A multimodal large-model dialogue system based on context awareness, characterized in that: Used to implement the context-aware multimodal large model dialogue method according to any one of claims 1 to 8; the system comprises: A multimodal input processing module, used for collecting a first dialogue message and preprocessing the first dialogue message; The context processing and dialogue management module is used to input the preprocessed first dialogue message into the context-aware dialogue management model to generate dialogue request information and perception feature information; the perception feature information includes: emotion vector information, analysis information based on historical dialogue information and user portrait context intention, and system hot word auxiliary information; the context processing and dialogue management module includes: Conversation memory, used to store user historical conversation data; Hot word product library, used to store social conversation hot words and product feature hot words The sentiment analysis unit is used to analyze user questions in combination with historical user conversation data and generate conversation request information and perception feature information; A dialogue routing module is used to determine the answer strategy based on the dialogue request information and the perception feature information; A generating module is used to generate a second dialogue message according to the answering strategy.

10. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the context-aware multimodal large model dialogue method as described in any one of claims 1 to 8.

Images