Multimodal-based dialogue reply method and device, electronic device and medium

Abstract

The embodiment of the application provides a kind of based on multi-modal dialogue reply method and device, electronic equipment and medium, belong to artificial intelligence technical field, it is applicable to the field of financial technology.The method comprises: by the preliminary fusion of question text feature and user information feature, obtain preliminary dialogue fusion feature, and the fusion of dialogue state feature and preliminary dialogue fusion feature, obtain target dialogue fusion feature, retrieve the dialogue reply knowledge feature corresponding to target dialogue fusion feature from business knowledge vector library, based on target dialogue fusion feature and dialogue reply knowledge feature, dialogue question text is generated for dialogue reply.This embodiment of the application can effectively capture the semantic features of the user's past dialogue through preliminary feature fusion and target feature fusion, and based on target dialogue fusion feature and dialogue reply knowledge feature retrieved from business knowledge vector, the dialogue reply of dialogue question text is generated, which improves the accuracy of dialogue reply.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the financial technology field, particularly to a multimodal dialogue response method, device, electronic device, and medium. Background Technology

[0002] Traditional dialogue response methods typically retrieve content from a question-and-answer knowledge base using similarity matching algorithms. Taking a financial scenario as an example, when a user asks online customer service about loan product interest rates, the algorithm searches a financial knowledge base containing information on financial products and interest rate changes for loan conditions. The resulting response might be the user's credit report, credit score, and proof of annual income. However, this method only matches financial knowledge stored in the knowledge base and ignores the contextual semantics of the dialogue, leading to low accuracy in dialogue responses. Therefore, improving the accuracy of dialogue responses has become a pressing issue. Summary of the Invention

[0003] The main objective of this application is to propose a multimodal dialogue response method, apparatus, electronic device, and medium, aiming to improve the accuracy of dialogue responses.

[0004] To achieve the above objectives, a first aspect of this application proposes a multimodal dialogue response method, the method comprising:

[0005] The dialogue question text of the target user is obtained, and the dialogue question text is subjected to feature extraction to obtain the dialogue question text features; wherein, the dialogue question text is the question text obtained at the current time.

[0006] The user information of the target user is obtained, and the user information is used to extract features to obtain user information features; wherein, the user information refers to structured user data information.

[0007] The dialogue question text features and the user information features are initially fused to obtain preliminary dialogue fusion features;

[0008] Obtain the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the past dialogue semantic features of the target user;

[0009] The dialogue state features are fused with the preliminary dialogue fusion features to obtain the target dialogue fusion features;

[0010] Dialogue response knowledge features are obtained by retrieving dialogue response from the target dialogue fusion features from a pre-built business knowledge vector library;

[0011] Based on the target dialogue fusion features and the dialogue response knowledge features, a dialogue response is generated from the dialogue question text to obtain the target dialogue response text.

[0012] In some embodiments, the step of generating a dialogue response text based on the target dialogue fusion features and the dialogue response knowledge features to obtain the target dialogue response text includes:

[0013] The dialogue response knowledge features and the target dialogue fusion features are aligned in feature space to obtain aligned knowledge features.

[0014] The aligned knowledge features are concatenated with the target dialogue fusion features to obtain the knowledge fusion features;

[0015] The dialogue response model is pre-trained based on the knowledge fusion feature indication to generate the response, thereby obtaining the target dialogue response text; wherein, the target dialogue response text is the response text to the dialogue question text.

[0016] In some embodiments, the step of generating the response based on the dialogue response model pre-trained with the knowledge fusion feature indication to obtain the target dialogue response text includes:

[0017] The dialogue response model, pre-trained based on the knowledge fusion feature indications, generates a response to obtain intermediate dialogue response text.

[0018] The dialogue question text is updated to obtain the updated dialogue question text.

[0019] The dialogue state features are updated based on the updated dialogue question text and the intermediate dialogue response text to obtain the updated dialogue state features.

[0020] The target dialogue fusion feature is updated based on the updated dialogue state feature to obtain the updated dialogue fusion feature, and the dialogue response knowledge feature is updated based on the updated dialogue state feature to obtain the updated dialogue response knowledge feature.

[0021] The intermediate dialogue response text is updated based on the updated dialogue fusion features and the updated dialogue response knowledge features to obtain the target dialogue response text.

[0022] In some embodiments, before generating a response using the dialogue response model pre-trained based on the knowledge fusion feature indications, the method further includes: training the dialogue response model, specifically including:

[0023] Obtain the original dialogue response model and training sample set; wherein, the training sample set includes training user information, multiple training dialogue question texts, real dialogue response texts corresponding to each training dialogue question text, real preliminary dialogue fusion features corresponding to each training dialogue question text and the training user information, and real dialogue response knowledge features corresponding to the real preliminary dialogue fusion features.

[0024] The training dialogue question text features are obtained by extracting features from multiple training dialogue question texts using the original dialogue response model.

[0025] Feature extraction is performed on the training user information to obtain training user information features;

[0026] The training dialogue question text features and the training user information features are fused to obtain preliminary training dialogue fusion features.

[0027] The training dialogue state features of the training dialogue question text features are obtained, and the training dialogue state features are fused with the initial training dialogue fusion features to obtain the training target dialogue fusion features.

[0028] Dialogue response retrieval is performed on the training target dialogue fusion features from the pre-built business knowledge vector library to obtain training dialogue response knowledge features;

[0029] Based on the training target dialogue fusion features and the training dialogue response knowledge features, the training dialogue question text is used to predict the dialogue response text, and the predicted dialogue response text is obtained.

[0030] The target loss values ​​of the training initial dialogue fusion features, the real initial dialogue fusion features, the training dialogue response knowledge features, the real dialogue response knowledge features, the predicted dialogue response text, and the real dialogue response text are calculated based on a preset loss function.

[0031] The model parameters of the original dialogue response model are updated based on the target loss value.

[0032] In some embodiments, the step of retrieving dialogue response knowledge features from the target dialogue fusion features in a pre-built business knowledge vector library to obtain dialogue response knowledge features includes:

[0033] The target dialogue fusion features are compressed to obtain compressed dialogue fusion features;

[0034] Obtain the business knowledge vectors from the business knowledge vector library, and perform feature space alignment processing between the compressed dialogue fusion features and the business knowledge vectors to obtain aligned fusion features;

[0035] The semantic similarity of the alignment fusion features and the business knowledge vector is calculated to obtain a similarity score;

[0036] Based on the similarity score, the business knowledge vector is used to filter dialogue response knowledge to obtain the dialogue response knowledge features.

[0037] In some embodiments, the preliminary fusion of the dialogue question text features and the user information features to obtain preliminary dialogue fusion features includes:

[0038] The dialogue question text features are spatially projected to obtain projected question text features, and the user information features are spatially projected to obtain projected user features; the projected question text features and the projected user features are located in the same vector dimension space;

[0039] The text features of the projection problem are normalized to obtain normalized text features, and the projection user features are normalized to obtain normalized user features.

[0040] The normalized text features and the normalized user features are concatenated to obtain the preliminary dialogue fusion features.

[0041] In some embodiments, the step of fusing the dialogue state features with the preliminary dialogue fusion features to obtain the target dialogue fusion features includes:

[0042] The dialogue state features are concatenated with the preliminary dialogue fusion features to obtain the concatenated dialogue fusion features;

[0043] The concatenated dialogue fusion features are subjected to self-attention processing to obtain the target dialogue fusion features.

[0044] To achieve the above objectives, a second aspect of this application provides a multimodal dialogue response device, the device comprising:

[0045] The question text feature extraction module is used to acquire the dialogue question text of the target user and extract features from the dialogue question text to obtain dialogue question text features; wherein, the dialogue question text is the question text acquired at the current time.

[0046] The user information feature extraction module is used to obtain the user information of the target user and extract features from the user information to obtain user information features; wherein, the user information refers to structured user data information;

[0047] The preliminary feature fusion module is used to perform preliminary feature fusion on the dialogue question text features and the user information features to obtain preliminary dialogue fusion features;

[0048] The dialogue state feature acquisition module is used to acquire the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the past dialogue semantic features of the target user.

[0049] The target feature fusion module is used to fuse the dialogue state features with the preliminary dialogue fusion features to obtain the target dialogue fusion features.

[0050] The business knowledge retrieval module is used to retrieve dialogue response features from the target dialogue fusion features in a pre-built business knowledge vector library to obtain dialogue response knowledge features.

[0051] The dialogue response generation module is used to generate a dialogue response based on the target dialogue fusion features and the dialogue response knowledge features, thereby obtaining the target dialogue response text.

[0052] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0053] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0054] The multimodal dialogue response method, apparatus, electronic device, and medium proposed in this application firstly achieve the fusion of dialogue features from different modalities by extracting features from the dialogue question text and user information respectively and performing preliminary feature fusion. This effectively captures detailed features containing user information, facilitating subsequent improvements in the accuracy of dialogue responses. Secondly, by identifying the dialogue state features of the dialogue question text features and fusing these features with the preliminary dialogue fusion features, the semantic features of the user in the previous round of dialogue can be effectively captured, effectively solving the dialogue semantic problem caused by neglecting context in traditional methods. Finally, dialogue response knowledge features are obtained by retrieving the target dialogue fusion features from a pre-built business knowledge vector library. This allows for the retrieval of dialogue response prompts or business processing instructions during the dialogue process. Based on the target dialogue fusion features and dialogue response knowledge features, dialogue responses are generated from the dialogue question text to obtain the target dialogue response text. This allows for dialogue response reasoning by combining knowledge from the knowledge base to automatically generate dialogue response content containing dialogue response instructions, significantly improving the accuracy of dialogue responses. Attached Figure Description

[0055] Figure 1 This is a flowchart of a multimodal dialogue response method provided in an embodiment of this application;

[0056] Figure 2 This is another flowchart of the multimodal dialogue response method provided in the embodiments of this application;

[0057] Figure 3 yes Figure 1 The flowchart of step S103 in the process;

[0058] Figure 4 yes Figure 1 The flowchart of step S105 in the process;

[0059] Figure 5 yes Figure 1 The flowchart of step S106 in the process;

[0060] Figure 6 yes Figure 1 The flowchart of step S107 in the process;

[0061] Figure 7 yes Figure 6 The flowchart of step S603 in the process;

[0062] Figure 8 This is a schematic diagram of the structure of the multimodal dialogue response device provided in the embodiments of this application;

[0063] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0067] First, let's analyze some of the terms used in this application:

[0068] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0069] This application provides a multimodal dialogue response method, apparatus, electronic device, and medium, aiming to improve the accuracy of dialogue responses.

[0070] The multimodal dialogue response method, apparatus, electronic device, and medium provided in this application are specifically described through the following embodiments. First, the multimodal dialogue response method in this application is described.

[0071] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0072] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0073] The multimodal dialogue response method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the multimodal dialogue response method, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] Figure 1 This is an optional flowchart of a multimodal dialogue response method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0076] Step S101: Obtain the dialogue question text of the target user, and extract features from the dialogue question text to obtain the dialogue question text features; wherein, the dialogue question text is the question text obtained at the current moment.

[0077] Step S102: Obtain user information of the target user and extract features from the user information to obtain user information features; where user information refers to structured user data information.

[0078] Step S103: Perform preliminary feature fusion on the dialogue question text features and user information features to obtain preliminary dialogue fusion features.

[0079] Step S104: Obtain the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the past dialogue semantic features of the target user.

[0080] Step S105: Perform feature fusion between the dialogue state features and the preliminary dialogue fusion features to obtain the target dialogue fusion features.

[0081] Step S106: Retrieve dialogue response features from the pre-built business knowledge vector library to obtain dialogue response knowledge features.

[0082] Step S107: Based on the target dialogue fusion features and dialogue response knowledge features, generate dialogue response text from the dialogue question text to obtain the target dialogue response text.

[0083] Steps S101 to S107 of this embodiment firstly involve feature extraction and preliminary feature fusion for the dialogue question text and user information, respectively, to achieve the fusion of dialogue features from different modalities and effectively capture detailed features containing user information, thus facilitating improved accuracy of subsequent dialogue responses. Secondly, by identifying the dialogue state features of the dialogue question text and fusing them with the preliminary dialogue fusion features, the semantic features of the user in the previous round of dialogue can be effectively captured, effectively solving the dialogue semantic problem caused by neglecting context in traditional methods. Finally, dialogue response knowledge features are retrieved from the pre-built business knowledge vector library for the target dialogue fusion features, enabling the retrieval of dialogue response prompts or business processing instructions during the dialogue process. Based on the target dialogue fusion features and dialogue response knowledge features, dialogue responses are generated from the dialogue question text to obtain the target dialogue response text. This allows for dialogue response reasoning by combining knowledge from the knowledge base to automatically generate dialogue response content containing dialogue response instructions, significantly improving the accuracy of dialogue responses.

[0084] In step S101 of some embodiments, specifically, the dialogue question text refers to the question text currently entered by the user.

[0085] For example, in financial applications, the dialogue question text could be, "I want to apply for a large loan, what are the requirements?"

[0086] Specifically, before extracting features from the dialogue question text using a pre-trained dialogue response model, it is necessary to train the dialogue response model.

[0087] Please see Figure 2 In some embodiments, the multimodal dialogue response method further includes, but is not limited to, steps S201 to S209:

[0088] Step S201: Obtain the original dialogue response model and training sample set; wherein, the training sample set includes training user information, multiple training dialogue question texts, real dialogue response texts corresponding to each training dialogue question text, real preliminary dialogue fusion features corresponding to each training dialogue question text and training user information, and real dialogue response knowledge features corresponding to the real preliminary dialogue fusion features.

[0089] Step S202: Extract features from multiple training dialogue question texts using the original dialogue response model to obtain the training dialogue question text features.

[0090] Step S203: Extract features from the training user information to obtain the training user information features.

[0091] Step S204: Perform preliminary training feature fusion on the training dialogue question text features and training user information features to obtain preliminary training dialogue fusion features.

[0092] Step S205: Obtain the training dialogue state features of the training dialogue question text features, and fuse the training dialogue state features with the initial training dialogue fusion features to obtain the training target dialogue fusion features.

[0093] Step S206: Retrieve dialogue responses from the pre-built business knowledge vector library to obtain training dialogue response knowledge features.

[0094] Step S207: Based on the training target dialogue fusion features and training dialogue response knowledge features, predict the dialogue response text of the training dialogue question text to obtain the predicted dialogue response text.

[0095] Step S208: Calculate the target loss values ​​of the training preliminary dialogue fusion features, the real preliminary dialogue fusion features, the training dialogue response knowledge features, the real dialogue response knowledge features, the predicted dialogue response text, and the real dialogue response text according to the preset loss function.

[0096] Step S209: Update the model parameters of the original dialogue response model based on the target loss value.

[0097] In step S201 of some embodiments, specifically, the dialogue response model is used to generate a dialogue response. This dialogue response model includes a question text feature extraction network, a user information feature extraction network, a preliminary feature fusion network, a dialogue state fusion network, a response knowledge retrieval network, and a dialogue response network. Specifically, the question text feature extraction network can be a finBERT network, used to capture the semantic and intent information of the question text; the user information feature extraction network is used to map structured user information into entity vectors; the preliminary feature fusion network is used to achieve multimodal feature fusion of dialogue question text features and user information features; the dialogue state fusion network is used to achieve the fusion of preliminary dialogue fusion features and dialogue semantic features; the business knowledge retrieval network is a dialogue response knowledge retrieval network implemented based on the RAG (Retrieval-Augmented Generation) retrieval mechanism, used to provide business risk warnings or business processing instructions for the dialogue question text; the dialogue response network can be a Transformer model, performing final dialogue response inference based on the final dialogue fusion features to generate the final dialogue response text.

[0098] Specifically, the training sample set is a dataset used to train the original dialogue response model, including but not limited to training user information, multiple training dialogue question texts, real dialogue response texts corresponding to each training dialogue question text, real preliminary dialogue fusion features corresponding to each training dialogue question text and training user information, and real dialogue response knowledge features corresponding to the real preliminary dialogue fusion features.

[0099] For example, in financial application scenarios, the training sample set may contain the personal information of training users (such as age, credit report, credit score, and loan status), the training dialogue question text may be "What is the annual interest rate of loan product G?", the real dialogue response text may be "The annual interest rate of loan product G is 20%", the real initial dialogue fusion features may be "loan product", "G", "annual interest rate" and "how much", the real initial dialogue fusion features may be "loan product", "G", "annual interest rate", "credit score", "credit report", etc., and the real dialogue response knowledge features may be "loan product", "G", "annual interest rate" and "20%", etc.

[0100] In step S202 of some embodiments, specifically, the training dialogue question text can be segmented to obtain training text tokens, and the training text tokens can be input into the text feature extraction network (such as the finBERT model) of the original dialogue response model for text token embedding operation in order to capture the semantic and syntactic information features of the training dialogue question text.

[0101] In step S203 of some embodiments, specifically, entity embedding operations can be performed on the training user information through a user information feature extraction network to convert the structured training user information into training entity vector data.

[0102] In step S204 of some embodiments, specifically, the training dialogue question text features and training user information features can be projected into a shared vector space of the same dimension through the MLP (Multilayer Perceptron) network layer in the preliminary feature fusion network, and the projected features are normalized by the L2 paradigm. The normalized training dialogue question text features and training user information features are then concatenated to obtain the training preliminary dialogue fusion features.

[0103] In step S205 of some embodiments, specifically, firstly, the vector dimension of the training dialogue state features can be updated through the dialogue state fusion network to obtain training dialogue state features with the same dimension as the initial training dialogue fusion features; secondly, the updated training dialogue state features and the initial training dialogue fusion features are concatenated to obtain the concatenated training dialogue fusion features; finally, the training dialogue fusion features are subjected to sequence-level self-attention processing to obtain the training target dialogue fusion features.

[0104] In step S206 of some embodiments, specifically, the training target dialogue fusion features can be pooled through a response knowledge retrieval network to achieve dimensionality reduction of the training target dialogue fusion features. The dimensionality-reduced training target dialogue fusion features are then mapped to a space with the same dimension as the training business knowledge vector. The similarity score between the mapped training target dialogue fusion features and the training business knowledge vector is calculated. Based on the similarity score, the training business knowledge vector that best matches the training target dialogue fusion features is selected as the training dialogue response knowledge feature.

[0105] In step S207 of some embodiments, specifically, the dialogue response knowledge features can be fused with the training target dialogue fusion features through the Transformer model to obtain knowledge fusion features, and the original dialogue response model can be instructed to perform dialogue response reasoning based on the knowledge fusion features to obtain the training dialogue response text.

[0106] In step S208 of some embodiments, specifically, the loss function includes a preliminary feature fusion loss sub-function, a dialogue feature fusion loss sub-function, and a dialogue response loss sub-function.

[0107] Specifically, the preliminary feature fusion loss function can be represented by the following formula:

[0108]

[0109] Among them, L fusion This represents the initial feature fusion loss function, where N represents the number of samples in the training sample set. This represents the text feature of the l-th training dialogue question. This represents the h-th training user information feature.

[0110] Specifically, in order to ensure the relevance between the initial dialogue fusion features and the training business knowledge vector of the knowledge document, and to prevent the introduction of irrelevant or redundant information, the training objective is to maximize the relevance between the initial dialogue fusion features and the training business knowledge vector, and to widen the gap between positive and negative values ​​in the dialogue state fusion network.

[0111] Specifically, the dialogue feature fusion loss function can be represented by the following formula:

[0112]

[0113] Among them, L relevance Let N represent the dialogue feature fusion loss function, and let N represent the number of samples in the training sample set. This represents the k-th preliminary dialogue fusion feature during training. This represents the i-th positive sample of the training business knowledge vector in the business knowledge vector library that corresponds to the initial training dialogue fusion features. Let j represent the negative sample of the training business knowledge vector in the business knowledge vector library corresponding to the initial dialogue fusion feature of the training.

[0114] Specifically, the dialogue response loss function can be represented by the following formula:

[0115]

[0116] Among them, L gen Let z represent the dialogue response loss function. input The training knowledge fusion feature, y, represents the fusion feature of training dialogue response knowledge features and training target dialogue knowledge features. t p(y) represents the predicted dialogue response text at the current time step t. t |y ＜t , z input ) represents the fusion feature z of training knowledge given the input. input Given all previous time steps from yt1 to yt-1, the probability of generating the predicted dialogue response text at the current time step t.

[0117] Specifically, the target loss value is a quantitative metric that measures the difference between the predicted dialogue response text generated by the model and the actual dialogue response text. The smaller the target loss value, the closer the prediction of the dialogue response model is to the real situation.

[0118] In this embodiment, the target loss values ​​of the training preliminary dialogue fusion features, the real preliminary dialogue fusion features, the training dialogue response knowledge features, the real dialogue response knowledge features, the predicted dialogue response text, and the real dialogue response text are calculated according to a preset loss function. This facilitates the subsequent updating of model parameters, reduces the prediction error of dialogue responses, and helps improve the accuracy of dialogue responses.

[0119] In step S209 of some embodiments, specifically, the model parameters of the original dialogue response model are updated based on the target loss value, and the process is repeated to extract features from multiple training dialogue question texts using the original dialogue response model until the original dialogue response model meets the preset training conditions, thereby obtaining a pre-trained dialogue response model.

[0120] Specifically, the preset training condition can be that the target loss value is less than a preset loss threshold. The preset loss threshold is determined based on the actual application scenario, and here it can be 0.8.

[0121] Specifically, if the original dialogue response model meets the preset training conditions, a pre-trained dialogue response model is obtained, which can be used for actual dialogue response tasks.

[0122] Through steps S201 to S209, the model learns how to extract comprehensive semantic features of the dialogue context from the training sample set through the parameter update process of the original dialogue response model. It also uses the business knowledge vector library to retrieve dialogue responses, enabling the model to integrate professional knowledge and ensure the professionalism of the dialogue responses. This provides users with financial anti-fraud tips and ensures the security of their business transactions. Furthermore, the model parameters are updated through the loss function, continuously optimizing the model's prediction accuracy and providing users with timely and accurate financial anti-fraud tips or business transaction instructions.

[0123] Furthermore, step S101 also includes feature extraction of the dialogue question text.

[0124] Specifically, dialogue question text features refer to the semantic information extracted from the user's current question text.

[0125] For example, in financial application scenarios, if a user asks, "I want to apply for a large loan, what are the requirements?", the dialogue question text features will extract key information such as "application", "large amount", "loan", "requirements" and "conditions".

[0126] Specifically, the dialogue question text is first segmented to obtain text tokens, and then the text tokens are input into the text feature extraction network (such as the finBERT model) of the pre-trained dialogue response model for text token embedding to capture the semantic and intent information of the question text.

[0127] Furthermore, the process of extracting features from dialogue question text can be represented by the following formula:

[0128]

[0129] in, This represents the vector output by the last network layer after the token has passed through finBERT. The dimension of the dialogue question text features can be seq_len×768, where seq_len is the length of the input text token sequence.

[0130] In this embodiment, by extracting features from the dialogue question text, the unstructured dialogue question text can be transformed into standardized features that can be processed by machines and the intention of recognizing features, providing a question text data foundation for subsequent multimodal feature fusion.

[0131] In step S102 of some embodiments, specifically, user information refers to the personal information of the user who asked the question, and the user information is structured user data information, which includes, but is not limited to, numerical data, discrete data, etc.

[0132] For example, a user's credit report, credit score, loan history, and annual income.

[0133] Specifically, user information features are information extracted from structured user data.

[0134] Specifically, user information can be embedded into entity data by using a user information feature extraction network to convert structured user information into entity vector data.

[0135] Specifically, user information characteristics can be represented by the following formula:

[0136]

[0137] Among them, E D This represents user information features, with dimension d. D D represents user information, and its dimension is L. D Entity is the process of converting structured user information into Entity Embedding.

[0138] Furthermore, for numerical user information, a fixed-dimensional vector representation Ed1 is obtained through linear transformation or nonlinear activation processing. For discrete user information, a trainable vector can be assigned to each discrete category index to obtain a vectorized representation Ed2. Ed1 and Ed2 are concatenated to obtain the user information features.

[0139] For example, structured user information might include a credit score of 4.5, no loan type, and an annual income of 30,000 yuan. The credit score of 4.5 can be mapped to a fixed-dimensional vector space of 10 using the linear transformation formula V = W x 4.5 + b. Here, V represents the feature information of 4.5, and W and b are the trainable weights and bias parameters of the model. For discrete data types, if there are multiple loan types, such as no loan, mortgage, and consumer loan, each category can be assigned a unique vector. Specifically, the loan type "no loan" can be assigned a 5-dimensional vector. This vector is continuously optimized during model training to better represent the meaning of the category in the financial context. After completing the above embedding operation, the 10-dimensional vectors of the credit score, annual income, and loan type are concatenated to obtain a user information feature with a total dimension of 25.

[0140] In this embodiment, feature extraction of user information can effectively capture the detailed features of user information, providing user data support for subsequent multimodal feature fusion.

[0141] Please see Figure 3 In some embodiments, step S103 includes, but is not limited to, steps S301 to S303:

[0142] Step S301: Spatial projection is performed on the dialogue question text features to obtain projected question text features, and spatial projection is performed on the user information features to obtain projected user features; the projected question text features and the projected user features are located in the same vector dimension space.

[0143] Step S302: Normalize the text features of the projection problem to obtain normalized text features, and normalize the projection user features to obtain normalized user features.

[0144] Step S303: The normalized text features and normalized user features are concatenated to obtain preliminary dialogue fusion features.

[0145] In step S301 of some embodiments, specifically, the projection problem text features and the projection user features are located in the same vector dimension space.

[0146] Specifically, the dialogue question text features and user information features can be projected into a shared vector space of the same dimension through the MLP network layer in the preliminary feature fusion network. Specifically, the dialogue question text features E... U And user information features E D The vectors are projected into vector spaces of dimensions seq_len×1024 and 1×1024. The initial feature fusion network is used to achieve multimodal feature fusion of dialogue question text features and user information features.

[0147] In this embodiment, by spatially projecting the dialogue question text features and user information features, feature vectors of different dimensions can be mapped to the same dimensional space, eliminating the dimensional differences between heterogeneous features and avoiding deviations in the subsequent fusion of different modal features due to vector scale mismatch.

[0148] In step S302 of some embodiments, specifically, normalized text features refer to the fact that the vector values ​​of each dimension of the projected text features follow a standard distribution with a mean of 0 and a variance of 1, and normalized user features refer to the fact that the vector values ​​of each dimension of the projected user features follow a standard distribution with a mean of 0 and a variance of 1.

[0149] Specifically, the projected features can be normalized using L2 normalization to obtain normalized features.

[0150] Specifically, normalized text features and normalized user features can be represented by the following formulas:

[0151] E′ U =L2(MLP(E) U ))∈R seq_len×1024

[0152] E′ D =L2(MLP(E) D ))∈R 1×1024

[0153] Among them, E′ U E′ represents the normalized text features. D E represents normalized user characteristics. U E represents the textual features of the dialogue question. D The seq_len×1024 represents the feature dimension of the normalized text feature, and 1×1024 represents the feature dimension of the normalized user feature.

[0154] In this embodiment, by normalizing the text features and user features of the projection problem, the differences in the vector numerical scale of different features can be eliminated. This effectively prevents the initial feature fusion network from overemphasizing features with large vector values, while preserving the relative proportional relationship between the dimensions of the vectors, which helps to improve the accuracy of subsequent feature fusion of different modalities.

[0155] In step S303 of some embodiments, specifically, the preliminary dialogue fusion features refer to the fused normalized text features and normalized user features.

[0156] Specifically, the initial dialogue fusion features can be represented by the following formula:

[0157] E = Concat(E′) U ,E′ D )∈R (seq_len+1)×1024

[0158] Where, E∈R (seq_len+1)×1024 This represents the initial dialogue fusion features, with a dimension of (seq_len+1)×1024, E′ U E′ represents the normalized text features. D This represents normalized user characteristics.

[0159] Through steps S301 to S303, the text features of the dialogue question and the user information features can be effectively fused. This process not only solves the problem of inconsistency between different feature dimensions, but also eliminates the influence of differences in feature dimensions through normalization. Finally, the question semantics and user personal information are combined through splicing operations, providing richer and more comprehensive feature information for subsequent dialogue processing, which helps to improve the accuracy of subsequent dialogue responses.

[0160] In step S104 of some embodiments, specifically, the dialogue state features refer to the contextual semantic features of the target user's past dialogues with the model, including but not limited to the text features of the user's previous question and response in the previous round of dialogue.

[0161] Furthermore, if the dialogue question text is the target user's first question text, then the dialogue state features can be the dialogue semantic features learned by the pre-trained dialogue response model.

[0162] For example, in a financial application scenario, if a user previously asked "Please check my account spending records", and the response text to that question was "Your account spending records are spending record 1, spending record 2, and spending record 3, etc.", then the semantic features of the dialogue would be the user's account spending records and spending record 1, spending record 2, and spending record 3.

[0163] In this embodiment, by obtaining the dialogue state features of the dialogue question text features, it is possible to effectively capture the semantic features of the user's past dialogues, which makes it easier to capture the risks that the user may have when applying for services.

[0164] Please see Figure 4 In some embodiments, step S105 includes, but is not limited to, steps S401 to S402:

[0165] Step S401: The dialogue state features are concatenated with the preliminary dialogue fusion features to obtain the concatenated dialogue fusion features.

[0166] Step S402: Perform self-attention processing on the spliced ​​dialogue fusion features to obtain the target dialogue fusion features.

[0167] In step S401 of some embodiments, specifically, splicing dialogue fusion features refers to the vector obtained by splicing dialogue state features and preliminary dialogue fusion features.

[0168] Specifically, before feature concatenation, the vector dimension of the dialogue state features can be updated through a dialogue state fusion network to obtain dialogue state features with the same dimension as the initial dialogue fusion features.

[0169] Specifically, the vector dimension update of the dialogue state features can be achieved using the following formula:

[0170] H′ n-1 =Linear(H n-1 )∈R seq_len_old×1024

[0171] Among them, H′ n-1 This represents the dialogue state features after the vector dimension is updated, where seq_len_old×1024 represents the vector dimension of the dialogue state features.

[0172] Furthermore, the splicing dialogue fusion features can be determined using the following formula:

[0173]

[0174] in, This represents the concatenated dialogue fusion features, with a vector dimension of (seq_len_old + seq_len + 1) × 1024. E represents the initial dialogue fusion feature, and H... n-1 This represents the dialogue state features after the vector dimension is updated, and Concat represents feature concatenation.

[0175] In this embodiment, by concatenating the dialogue state features with the initial dialogue fusion features, the semantic features of the user's past dialogues can be effectively captured. This effectively solves the dialogue semantic problem caused by the traditional method ignoring the context. Furthermore, it integrates the risk features that the user may have when applying for business services, which helps to improve the accuracy of subsequent dialogue responses.

[0176] In step S402 of some embodiments, specifically, the dialogue state fusion network further includes multi-layer Attention Blocks, which input the spliced ​​dialogue fusion features into the Attention Blocks to perform sequence-level self-attention processing on the spliced ​​dialogue fusion features to obtain the target dialogue fusion features.

[0177] Specifically, the target dialogue fusion features can be represented by the following formula:

[0178]

[0179] Among them, z n This represents the target dialogue fusion feature of the nth round of dialogue, with a vector dimension of (seq_len_old + seq_len + 1) × 1024. This indicates the characteristics of splicing dialogue fusion.

[0180] Through steps S401 to S402, the dialogue state features and the initial dialogue fusion features can be effectively fused. This process not only preserves the complete past dialogue information and current question information of the user by splicing, but also further highlights key information through self-attention processing, further captures the risk features that the user may have, and effectively solves the dialogue semantic problem caused by the traditional method ignoring the context, which helps to improve the accuracy of subsequent dialogue responses.

[0181] Please see Figure 5 In some embodiments, step S106 includes, but is not limited to, steps S501 to S504:

[0182] Step S501: Perform feature compression on the target dialogue fusion features to obtain compressed dialogue fusion features.

[0183] Step S502: Obtain the business knowledge vectors from the business knowledge vector library, and perform feature space alignment processing on the compressed dialogue fusion features and the business knowledge vectors to obtain aligned fusion features.

[0184] Step S503: Perform semantic similarity calculation on the aligned fusion features and business knowledge vectors to obtain a similarity score.

[0185] Step S504: Based on the similarity score, perform dialogue response knowledge filtering on the business knowledge vector to obtain dialogue response knowledge features.

[0186] In step S501 of some embodiments, specifically, compressed dialogue fusion features refer to fusion features after dimensionality reduction.

[0187] Specifically, the target dialogue fusion features can be pooled using a response knowledge retrieval network to achieve dimensionality reduction of the target dialogue fusion features.

[0188] Specifically, the compressed dialogue fusion features can be determined using the following formula:

[0189]

[0190] in, This represents compressed dialogue fusion features, with a vector dimension of 1x1024. `Pool` indicates pooling processing, and `z` represents the vector size. n This indicates the target dialogue fusion feature.

[0191] In this embodiment, by performing feature compression on the target dialogue fusion features, the high-dimensional target dialogue fusion features can be reduced to low-dimensional feature vectors while retaining key information. This ensures the accuracy of the dialogue response retrieval network and improves the efficiency of subsequent dialogue response retrieval.

[0192] In step S502 of some embodiments, specifically, the business knowledge vector library is a pre-built knowledge base that stores business knowledge feature vectors related to business processing.

[0193] Specifically, business knowledge vectors refer to the knowledge vectors stored in the business knowledge vector library.

[0194] For example, in financial application scenarios, the business knowledge vector library can be a financial business knowledge vector library, which includes, but is not limited to, knowledge feature vectors such as loan business conditions, transfer business, transfer purpose, financial product application conditions, and financial business anti-fraud.

[0195] Specifically, the acquired financial knowledge documents are integrated to form a complete financial knowledge document, which is then sliced ​​into multiple knowledge document chunks. The finBERT model is used to embed the text into each chunk, resulting in an embedding vector for each chunk. (i.e., business knowledge vectors) are stored in the Vector Store (i.e., the business knowledge vector library). Among them, E... C This represents a business knowledge vector, with a vector dimension of 1×d. C .

[0196] Specifically, aligned fusion features refer to mapping compressed dialogue fusion features into a space with the same dimension as the business knowledge vector.

[0197] Specifically, the alignment and blending features can be represented by the following formula:

[0198]

[0199] Where, z′ n This represents the alignment and fusion feature, with a vector dimension of 1×d. C tanh represents the activation function. This indicates compressed dialogue fusion features. The weight parameters represent the compressed dialogue fusion feature projection, with a vector dimension of d. h ×d C , The bias matrix represents the projection of compressed dialogue fusion features, with a vector dimension of d. h ×1.

[0200] In step S503 of some embodiments, specifically, the similarity score refers to the semantic similarity score between the aligned fusion feature and the business knowledge vector.

[0201] Specifically, semantic similarity can be calculated using the following formula:

[0202]

[0203] Where s represents the similarity score, z′ n E represents the alignment and fusion feature. C This represents a business knowledge vector.

[0204] In step S504 of some embodiments, specifically, the dialogue response knowledge feature refers to the dialogue response knowledge feature related to the dialogue question text feature.

[0205] For example, in financial application scenarios, if the knowledge features of the dialogue response are application, large amount, loan, need, and conditions, then the knowledge features of the dialogue response could be proof of income, credit report, credit score, or reaching 600 points or above.

[0206] Specifically, the most matching dialogue response knowledge features are selected based on the similarity score s. Specifically, the top 5 knowledge chunks with the highest scores are selected from the chunks sorted in descending order of similarity scores and then concatenated to obtain the dialogue response knowledge features.

[0207] In this embodiment, the dialogue response knowledge is filtered based on the similarity score of the business knowledge vector. This can quantify the similarity between the user's question text and each knowledge item in the business knowledge vector library, so as to filter out the knowledge items most relevant to the user's question text, thereby improving the accuracy and efficiency of business knowledge retrieval.

[0208] Through steps S501 to S504, the business knowledge features most relevant to the dialogue question text can be retrieved from the pre-built business knowledge vector library. This process not only reduces the amount of computation through feature compression, but also improves the accuracy and efficiency of knowledge retrieval through feature space alignment and semantic similarity calculation, which helps to provide business risk prompts or business processing instructions in the subsequent dialogue between the user and the model.

[0209] Please see Figure 6 In some embodiments, step S107 includes, but is not limited to, steps S601 to S603:

[0210] Step S601: Align the dialogue response knowledge features with the target dialogue fusion features in the feature space to obtain aligned knowledge features.

[0211] Step S602: The aligned knowledge features and the target dialogue fusion features are concatenated to obtain the knowledge fusion features.

[0212] Step S603: Generate a response based on the pre-trained dialogue response model using knowledge fusion feature indicators to obtain the target dialogue response text; wherein, the target dialogue response text is the response text to the dialogue question text.

[0213] In step S601 of some embodiments, specifically, aligned knowledge features can be obtained by projecting the dialogue response knowledge features into a vector space with the same vector dimension as the target dialogue fusion features.

[0214] Specifically, the alignment knowledge characteristics can be represented by the following formula:

[0215]

[0216] in, This represents the q-th alignment knowledge feature, with a vector dimension of 1×1024. This represents the q-th business knowledge vector; The projection weight parameters represent the knowledge features of the dialogue response, with a vector dimension of d. h ×d C , The projection bias matrix represents the knowledge features of the dialogue response, with a vector dimension of d. h ×1.

[0217] In step S602 of some embodiments, specifically, knowledge fusion features can be formed by connecting the aligned knowledge features and the target dialogue fusion features together in vector order.

[0218] Specifically, the characteristics of knowledge fusion can be represented by the following formula:

[0219]

[0220] Where, z′ Input The vector dimension represents the knowledge fusion feature, which is (seq_len_old+seq_len+5)×1024. q is the number of selected business knowledge vectors, i.e., the number of chunks is 4.

[0221] Please see Figure 7 In some embodiments, step S603 includes, but is not limited to, steps S701 to S705:

[0222] Step S701: Generate a response based on the pre-trained dialogue response model using knowledge fusion feature indicators to obtain the intermediate dialogue response text.

[0223] Step S702: Perform update detection on the dialogue question text to obtain the updated dialogue question text.

[0224] Step S703: Update the dialogue state features based on the updated dialogue question text and the intermediate dialogue response text to obtain the updated dialogue state features.

[0225] Step S704: Update the target dialogue fusion features based on the updated dialogue state features to obtain updated dialogue fusion features, and update the dialogue response knowledge features based on the updated dialogue state features to obtain updated dialogue response knowledge features.

[0226] Step S705: Update the intermediate dialogue response text based on the updated dialogue fusion features and the updated dialogue response knowledge features to obtain the target dialogue response text.

[0227] In step S701 of some embodiments, specifically, the intermediate dialogue response text is a preliminary dialogue response text in response to the dialogue question text.

[0228] Specifically, knowledge fusion features are input into the Transformer-type language model of the dialogue response network, which generates possible dialogue response texts in an autoregressive manner.

[0229] For example, in a financial application scenario, if the dialogue question text is the first round of dialogue text proposed by the user, specifically asking what conditions are required to apply for a large loan, and the user information is a credit score of 4.5, loan type: none, annual income: 100,000, and the dialogue state features are the training dialogue semantic features learned by the dialogue response model, specifically, the user's inquiry, which does not involve any abnormal business behavior at the moment, then the dialogue response knowledge features retrieved from the business knowledge vector library can be income proof, credit report, and credit score. Then, by combining the semantic features of "I want to apply for a large loan, what conditions are required," "credit score: 4.5," "loan type: none," "annual income: 100,000," "no abnormal business behavior," "income proof," "credit report," and "credit score" through the dialogue response network, the preliminary dialogue response text can be inferred as: "Applying for a loan requires providing income proof and a credit report, and the credit score must be above 600."

[0230] In step S702 of some embodiments, specifically, the dialogue question text is updated to a new dialogue question text proposed by the user.

[0231] Specifically, since users may engage in multiple rounds of dialogue during their interaction with the model, the updated dialogue question text can be the text of the dialogue question posed by the user in a new round.

[0232] For example, in financial applications, the updated dialogue question text could be something like, "I have no proof of income, what should I do?"

[0233] In step S703 of some embodiments, specifically, updating the dialogue state features refers to the semantic features that include the dialogue question text raised by the target user and the intermediate dialogue response text.

[0234] For example, in financial application scenarios, updating dialogue status features can include semantic features such as "I want to apply for a large loan, what conditions are required," "target user information: credit score: 4.5, loan type: none, annual income: 100,000, no abnormal business behavior, income certificate, credit report, credit score," and "applying for a loan requires providing income certificate and credit report, and the credit score must reach 600 or above."

[0235] In step S704 of some embodiments, specifically, updating the dialogue fusion feature refers to the feature obtained by combining the latest dialogue state feature with the target dialogue fusion feature. That is, updating the dialogue fusion feature integrates the target user's historical dialogue semantic information and the current target dialogue fusion feature, so that the dialogue response model can more comprehensively understand the user's intent and needs.

[0236] For example, in financial applications, if a user mentions that they have no proof of income and their credit score is low, the model can identify that the user may be experiencing financial difficulties by using the semantic features of the loan conditions, proof of income, and credit report from the user's previous inquiry, which require a credit score of 600 or higher.

[0237] Specifically, updating dialogue response knowledge features means searching for updated business knowledge vectors that match the updated dialogue question text from the business knowledge vector library based on the updated dialogue state features.

[0238] In step S705 of some embodiments, specifically, the target dialogue response text is the final dialogue response text generated for updating the dialogue question text.

[0239] For example, if the updated dialogue question text is "I have no proof of income, what should I do?", and the user has a low credit score and insufficient account balance, and the semantic features of the user's previous question regarding loan conditions, proof of income, and credit report, and a credit score of 600 or above, the target dialogue response text can be generated as "You can provide other forms of collateral, or try applying for a small loan."

[0240] In one embodiment of this application, if the user further asks, "Can I transfer some money to your internal account as a deposit?", and the dialogue response model further identifies a low user credit score, insufficient account balance, and recent frequent transfers, and considering the information from the first two rounds of dialogue and the dialogue state characteristics of the response text, indicating a potential financial risk, the final generated dialogue response text could be: "Please note, we will not ask you to transfer any deposit. This is a common scam; please do not transfer money to avoid losses."

[0241] Through steps S701 to S705, as the depth of user inquiries about financial business increases, the contextual semantics of the user's past dialogues are integrated to automatically adjust the response text. Combined with real-time optimization of knowledge retrieval, the dialogue response reasoning can be dynamically combined with knowledge in the knowledge base to automatically generate dialogue response content that provides business risk warnings or business processing instructions when the user has financial risk characteristics. This not only improves the efficiency and accuracy of dialogue responses but also ensures the security of users when handling financial business.

[0242] Through steps S601 to S603, by aligning the feature space, concatenating features, and generating dialogue responses, the effective fusion of dialogue response knowledge features and target dialogue fusion features is achieved. This not only improves the expressive power of the features but also enhances the dialogue response generation capability of the dialogue response model, thereby generating more accurate and professional dialogue response text.

[0243] This application first extracts and performs preliminary feature fusion on the dialogue question text and user information respectively, enabling the fusion of dialogue features from different modalities and effectively capturing detailed features containing user information, facilitating subsequent risk assessment for financial business processing. Second, by identifying the dialogue state features of the dialogue question text and fusing them with the preliminary dialogue fusion features, it effectively captures the semantic features of the user in the previous round of dialogue, effectively solving the dialogue semantic problem caused by the neglect of context in traditional methods. Finally, it retrieves dialogue response knowledge features from a pre-built business knowledge vector library based on the target dialogue fusion features, enabling the retrieval of relevant business risk warnings or business processing instructions when the user's business processing exhibits risk characteristics during the dialogue process. Based on the target dialogue fusion features and dialogue response knowledge features, it generates a dialogue response text from the dialogue question text, obtaining the target dialogue response text. This text can be combined with knowledge from the knowledge base for dialogue response reasoning to automatically generate dialogue response content containing business risk warnings or business processing instructions, improving the accuracy of dialogue responses and effectively reducing the risks for users during financial business processing.

[0244] Please see Figure 8 This application also provides a multimodal dialogue response device that can implement the above-mentioned multimodal dialogue response method. The device includes:

[0245] The question text feature extraction module is used to acquire the dialogue question text of the target user and extract features from the dialogue question text to obtain the dialogue question text features; wherein, the dialogue question text is the question text acquired at the current moment;

[0246] The user information feature extraction module is used to obtain the user information of the target user and extract features from the user information to obtain user information features; where user information refers to structured user data information.

[0247] The preliminary feature fusion module is used to perform preliminary feature fusion of dialogue question text features and user information features to obtain preliminary dialogue fusion features.

[0248] The dialogue state feature acquisition module is used to acquire the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the semantic features of the target user's past dialogues.

[0249] The target feature fusion module is used to fuse dialogue state features with preliminary dialogue fusion features to obtain target dialogue fusion features.

[0250] The business knowledge retrieval module is used to retrieve dialogue response features from the target dialogue fusion features in a pre-built business knowledge vector library to obtain dialogue response knowledge features.

[0251] The dialogue response generation module is used to generate dialogue responses from dialogue question text based on the target dialogue fusion features and dialogue response knowledge features, thus obtaining the target dialogue response text.

[0252] The specific implementation of this multimodal dialogue response device is basically the same as the specific embodiment of the multimodal dialogue response method described above, and will not be repeated here.

[0253] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned multimodal dialogue response method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0254] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0255] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0256] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the processing system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 to execute the multimodal dialogue response method of the embodiments of this application.

[0257] The input / output interface 903 is used to implement information input and output;

[0258] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0259] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0260] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0261] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal dialogue response method.

[0262] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0263] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0264] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0265] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0266] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0267] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0268] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0269] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0270] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0271] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0272] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0273] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multimodal dialogue response method, characterized in that, The method includes: The dialogue question text of the target user is obtained, and the dialogue question text is subjected to feature extraction to obtain the dialogue question text features; wherein, the dialogue question text is the question text obtained at the current time. The user information of the target user is obtained, and the user information is used to extract features to obtain user information features; wherein, the user information refers to structured user data information. The dialogue question text features and the user information features are initially fused to obtain preliminary dialogue fusion features; Obtain the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the past dialogue semantic features of the target user; The dialogue state features are fused with the preliminary dialogue fusion features to obtain the target dialogue fusion features; Dialogue response knowledge features are obtained by retrieving dialogue response from the target dialogue fusion features from a pre-built business knowledge vector library; Based on the target dialogue fusion features and the dialogue response knowledge features, a dialogue response is generated from the dialogue question text to obtain the target dialogue response text. The step of generating a dialogue response based on the target dialogue fusion features and the dialogue response knowledge features to obtain the target dialogue response text includes: The dialogue response knowledge features and the target dialogue fusion features are aligned in feature space to obtain aligned knowledge features. The aligned knowledge features are concatenated with the target dialogue fusion features to obtain the knowledge fusion features; The dialogue response model is pre-trained based on the knowledge fusion feature indication to generate the response, thereby obtaining the target dialogue response text; wherein, the target dialogue response text is the response text to the dialogue question text; The dialogue response model, pre-trained based on the knowledge fusion feature indications, generates the target dialogue response text, including: The dialogue response model, pre-trained based on the knowledge fusion feature indications, generates a response to obtain intermediate dialogue response text. The dialogue question text is updated to obtain the updated dialogue question text. The dialogue state features are updated based on the updated dialogue question text and the intermediate dialogue response text to obtain the updated dialogue state features. The target dialogue fusion feature is updated based on the updated dialogue state feature to obtain the updated dialogue fusion feature, and the dialogue response knowledge feature is updated based on the updated dialogue state feature to obtain the updated dialogue response knowledge feature. The intermediate dialogue response text is updated based on the updated dialogue fusion features and the updated dialogue response knowledge features to obtain the target dialogue response text.

2. The method according to claim 1, characterized in that, Before generating a response using the dialogue response model pre-trained based on the knowledge fusion feature indications, the method further includes: training the dialogue response model, specifically including: Obtain the original dialogue response model and training sample set; wherein, the training sample set includes training user information, multiple training dialogue question texts, real dialogue response texts corresponding to each training dialogue question text, real preliminary dialogue fusion features corresponding to each training dialogue question text and the training user information, and real dialogue response knowledge features corresponding to the real preliminary dialogue fusion features. The training dialogue question text features are obtained by extracting features from multiple training dialogue question texts using the original dialogue response model. Feature extraction is performed on the training user information to obtain training user information features; The training dialogue question text features and the training user information features are fused to obtain preliminary training dialogue fusion features. The training dialogue state features of the training dialogue question text features are obtained, and the training dialogue state features are fused with the initial training dialogue fusion features to obtain the training target dialogue fusion features. Dialogue response retrieval is performed on the training target dialogue fusion features from the pre-built business knowledge vector library to obtain training dialogue response knowledge features; Based on the training target dialogue fusion features and the training dialogue response knowledge features, the training dialogue question text is used to predict the dialogue response text, and the predicted dialogue response text is obtained. The target loss values ​​of the training initial dialogue fusion features, the real initial dialogue fusion features, the training dialogue response knowledge features, the real dialogue response knowledge features, the predicted dialogue response text, and the real dialogue response text are calculated based on a preset loss function. The model parameters of the original dialogue response model are updated based on the target loss value.

3. The method according to claim 1, characterized in that, The step of retrieving dialogue response knowledge features from the pre-built business knowledge vector library based on the target dialogue fusion features to obtain dialogue response knowledge features includes: The target dialogue fusion features are compressed to obtain compressed dialogue fusion features; Obtain the business knowledge vectors from the business knowledge vector library, and perform feature space alignment processing between the compressed dialogue fusion features and the business knowledge vectors to obtain aligned fusion features; The semantic similarity of the alignment fusion features and the business knowledge vector is calculated to obtain a similarity score; Based on the similarity score, the business knowledge vector is used to filter dialogue response knowledge to obtain the dialogue response knowledge features.

4. The method according to claim 1, characterized in that, The preliminary fusion of the dialogue question text features and the user information features to obtain preliminary dialogue fusion features includes: The dialogue question text features are spatially projected to obtain projected question text features, and the user information features are spatially projected to obtain projected user features; the projected question text features and the projected user features are located in the same vector dimension space; The text features of the projection problem are normalized to obtain normalized text features, and the projection user features are normalized to obtain normalized user features. The normalized text features and the normalized user features are concatenated to obtain the preliminary dialogue fusion features.

5. The method according to any one of claims 1 to 4, characterized in that, The step of fusing the dialogue state features with the preliminary dialogue fusion features to obtain the target dialogue fusion features includes: The dialogue state features are concatenated with the preliminary dialogue fusion features to obtain the concatenated dialogue fusion features; The concatenated dialogue fusion features are subjected to self-attention processing to obtain the target dialogue fusion features.

6. A multimodal dialogue response device, characterized in that, The device includes: The question text feature extraction module is used to acquire the dialogue question text of the target user and extract features from the dialogue question text to obtain dialogue question text features; wherein, the dialogue question text is the question text acquired at the current time. The user information feature extraction module is used to obtain the user information of the target user and extract features from the user information to obtain user information features; wherein, the user information refers to structured user data information; The preliminary feature fusion module is used to perform preliminary feature fusion on the dialogue question text features and the user information features to obtain preliminary dialogue fusion features; The dialogue state feature acquisition module is used to acquire the dialogue state features of the dialogue question text features; wherein, the dialogue state features are used to characterize the past dialogue semantic features of the target user. The target feature fusion module is used to fuse the dialogue state features with the preliminary dialogue fusion features to obtain the target dialogue fusion features. The business knowledge retrieval module is used to retrieve dialogue response features from the target dialogue fusion features in a pre-built business knowledge vector library to obtain dialogue response knowledge features. The dialogue response generation module is used to generate a dialogue response based on the target dialogue fusion features and the dialogue response knowledge features to obtain the target dialogue response text. The step of generating a dialogue response based on the target dialogue fusion features and the dialogue response knowledge features to obtain the target dialogue response text includes: The dialogue response knowledge features and the target dialogue fusion features are aligned in feature space to obtain aligned knowledge features. The aligned knowledge features are concatenated with the target dialogue fusion features to obtain the knowledge fusion features; The dialogue response model is pre-trained based on the knowledge fusion feature indication to generate the response, thereby obtaining the target dialogue response text; wherein, the target dialogue response text is the response text to the dialogue question text; The dialogue response model, pre-trained based on the knowledge fusion feature indications, generates the target dialogue response text, including: The dialogue response model, pre-trained based on the knowledge fusion feature indications, generates a response to obtain intermediate dialogue response text. The dialogue question text is updated to obtain the updated dialogue question text. The dialogue state features are updated based on the updated dialogue question text and the intermediate dialogue response text to obtain the updated dialogue state features. The target dialogue fusion feature is updated based on the updated dialogue state feature to obtain the updated dialogue fusion feature, and the dialogue response knowledge feature is updated based on the updated dialogue state feature to obtain the updated dialogue response knowledge feature. The intermediate dialogue response text is updated based on the updated dialogue fusion features and the updated dialogue response knowledge features to obtain the target dialogue response text.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal dialogue response method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal dialogue response method according to any one of claims 1 to 5.

Images