Session response method and apparatus, electronic device and storage medium

Abstract

Provided are a session response method and apparatus, an electronic device and a storage medium. The method comprises: acquiring multi-modal session information arranged according to a session time sequence, wherein the multi-modal session information comprises first modal information and second modal information (S110); performing multi-modal semantic representation on at least two modal features in the first modal information to obtain a first representation vector (S120); performing semantic representation of a corresponding modal on the second modal information to obtain a second representation vector (S130); respectively inputting the first representation vector, the second representation vector and a corresponding session time sequence into a session matching model to obtain a matching result (S140); and responding to the multi-modal session information according to the matching result (S150).

Description

Conversation answering method and device, electronic device, and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202310913820.9, filed on July 24, 2023, entitled “Conversational Response Method and Device, Electronic Device, Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of information processing technology, and in particular to a conversation answering method and device, an electronic device, and a storage medium. Background Art

[0004] Intelligent customer service, a technology based on natural language processing, has been widely adopted across major internet platforms in recent years. Related technologies primarily rely on natural language processing, specifically understanding and responding to user input text. However, with the diversification of user needs, user input has become multimodal, and single-text understanding cannot fully grasp the user's context and intent, resulting in low response accuracy and a poor user experience.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0006] Summary of the Invention

[0007] According to a first aspect of the present disclosure, a conversation response method is provided, comprising: obtaining multimodal conversation information arranged according to conversation time sequence, the multimodal conversation information including first modal information and second modal information; performing multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector; performing semantic representation of the second modal information in a corresponding modality to obtain a second representation vector; inputting the first representation vector and the second representation vector and the corresponding conversation time sequence into a conversation matching model to obtain a matching result; and responding to the multimodal conversation information based on the matching result.

[0008] Optionally, a multimodal semantic representation is performed on at least two modal features in the first modal information, including: extracting text content and text position information in the first modal information; dividing the first modal information into blocks to obtain block information and block position coordinates; vectorizing the text content, text position information, block information, and block position coordinates respectively; and performing a multimodal semantic representation on the vectorized representation result through a multimodal semantic representation model to obtain a first representation vector.

[0009] Optionally, the text content, text position information, block information, and block position coordinates are respectively vectorized, including: embedding and mapping the text content, text position information, and block position coordinates respectively; and linearly mapping the block information.

[0010] Optionally, the multimodal semantic representation model includes a first encoding model, a second encoding model and a feature fusion model, which performs multimodal semantic representation on the vectorized representation result, including: encoding the vector representation result of the block information and the vector representation result of the block position coordinates respectively through the first encoding model to obtain a first encoding vector; encoding the vector representation of the text content and the vector representation of the text position information respectively through the second encoding model to obtain a second encoding vector; and performing feature fusion on the concatenated vector of the first encoding vector and the second encoding vector through the feature fusion model to obtain a first representation vector; wherein, the first encoding model and the second encoding model are obtained through a pre-training process based on contrastive learning.

[0011] Optionally, the method also includes: obtaining a business corpus of the target business, the business corpus including at least two first modal corpora; taking the image features and text features of one first modal corpus as positive examples, and taking the image features of one first modal corpus and the text features of another first modal corpus as negative examples to construct a first training sample; using the first training sample to perform comparative learning training on the first encoding model and the second encoding model to obtain a pre-trained multimodal encoding model.

[0012] Optionally, the first training sample is used to perform comparative learning training on the first encoding model and the second encoding model, including: inputting image features corresponding to positive examples and negative examples in the first training sample into the first encoding model, and outputting an image representation vector; inputting text features corresponding to positive examples and negative examples in the first training sample into the second encoding model, and outputting a text representation vector; calculating the contrast loss based on the image representation vector and the text representation vector; and training the first encoding model and the second encoding model based on the contrast loss to obtain a pre-trained multimodal encoding model.

[0013] Optionally, the second modal information is semantically represented in the corresponding modality, including: semantically encoding the second modal information through a semantic coding model to obtain a second representation vector, where the semantic coding model is obtained by training a pre-trained third coding model.

[0014] Optionally, the method also includes: taking each multimodal conversation corpus in the business corpus as a second training sample; retrieving the label of the second training sample in the retrieval knowledge base, the retrieval knowledge base is constructed based on the business corpus, the retrieval knowledge base includes the domain dimension, the intention dimension within the domain and the question-answer pair dimension under the intention, and the label includes the label under each dimension; using the second training sample and the corresponding label, the pre-trained first encoding model, the second encoding model, the third encoding model and the feature fusion model, and the conversation matching model are trained respectively.

[0015] Optionally, the session matching model includes a context representation model and multiple prediction heads, one prediction head corresponds to a dimension in the retrieval knowledge base, and obtains a matching result, including: using the context representation model, performing feature encoding based on a multi-head attention mechanism on the first representation vector and the second representation vector and the corresponding session timing to obtain a third representation vector; inputting the third representation vector into each prediction head respectively to obtain matching results corresponding to each dimension.

[0016] Optionally, obtaining matching results for each dimension includes: using each prediction head to perform full connection processing and nonlinear activation processing on the third representation vector to obtain the prediction probability score corresponding to each prediction head; determining the prediction result with the prediction probability score in the top n digits in each prediction head as the matching result, where n is a positive integer.

[0017] Optionally, the method also includes: performing keyword matching on the text content and the second modal information in the domain keyword library; performing keyword matching on the text content and the second modal information in the intent keyword library; wherein the domain keyword library and the intent keyword library are respectively constructed based on the knowledge information of the corresponding dimensions of the retrieval knowledge base.

[0018] Based on the matching results, the multimodal conversation information is responded to, including: according to the matching results of each prediction head and the keyword matching results, the matching results of the prediction heads corresponding to the question-answer pair dimensions are rearranged to obtain a rearrangement score; and the question-answer pair with a rearrangement score greater than a preset threshold is used as the response content for the conversation response.

[0019] According to a second aspect of the present disclosure, a conversation response device is provided, comprising: an acquisition module, a first representation module, a second representation module, a conversation matching module, and a response module, wherein: the acquisition module is configured to acquire multimodal conversation information arranged in conversation time sequence, the multimodal conversation information including first modal information and second modal information; the first representation module is configured to perform multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector; the second representation module is configured to perform semantic representation of the corresponding modality on the second modal information to obtain a second representation vector; the conversation matching module is configured to input the first representation vector and the second representation vector and the corresponding conversation time sequence into a conversation matching model respectively to obtain a matching result; and the response module is configured to respond to the multimodal conversation information based on the matching result.

[0020] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

[0021] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to execute the method of any of the above embodiments.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0024] FIG1 schematically shows a flow chart of a conversation answering method according to an embodiment of the present disclosure.

[0025] FIG2 schematically shows a flowchart of a multimodal characterization process of first modality information according to an embodiment of the present disclosure.

[0026] FIG3 schematically shows a flow chart of a comparative learning training process according to an embodiment of the present disclosure.

[0027] FIG4 schematically shows a schematic diagram of a cascade search knowledge base according to an embodiment of the present disclosure.

[0028] FIG5 schematically shows a flow chart of a keyword matching process of a keyword library according to an embodiment of the present disclosure.

[0029] FIG6 schematically shows a flowchart of a processing procedure of a conversation answering process according to an embodiment of the present disclosure.

[0030] FIG7 schematically shows a structural block diagram of a conversation answering device according to an embodiment of the present disclosure.

[0031] FIG8 shows a block diagram of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0033] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The example disclosed herein provides an exemplary system diagram of an application scenario of a conversation response method and apparatus, the system including a terminal and a server. The method of this embodiment can be applied to a server. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, or a node in a blockchain.

[0035] For example, during the customer service process, the user inputs multimodal conversation information on the client. The server can perform multimodal semantic representation on at least two modal features in the first modal information based on the multimodal conversation information to obtain a first representation vector; perform semantic representation of the second modal information in the corresponding modality to obtain a second representation vector; input the first representation vector and the second representation vector and the corresponding conversation timing into the conversation matching model respectively to obtain a matching result; determine the response content based on the matching result, and send the response content to the user terminal.

[0036] The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, in-vehicle device, etc., but is not limited thereto. When the conversation response method provided in this embodiment is implemented through interaction between the terminal and the server, the terminal and the server may be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any restrictions thereto.

[0037] The conversation response method provided in the embodiment of the present disclosure may be executed on a server, and accordingly, the conversation response device is generally provided in the server.

[0038] The conversation response method disclosed in the embodiments of this specification is introduced below in conjunction with specific embodiments.

[0039] Referring to FIG1 , a conversation response method according to an exemplary embodiment of the present disclosure can be applied to various intelligent customer service systems and may include the following steps:

[0040] Step S110: Acquire multimodal session information arranged in session time sequence, where the multimodal session information includes first modal information and second modal information.

[0041] Step S120: Perform multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector.

[0042] Step S130 , performing semantic representation of the corresponding modality on the second modal information to obtain a second representation vector.

[0043] Step S140 : Input the first characterization vector, the second characterization vector, and the corresponding session time sequence into a session matching model to obtain a matching result.

[0044] Step S150: respond to the multimodal conversation information according to the matching result.

[0045] In the conversation response method provided in this example embodiment, on the one hand, semantic representation can be performed on each of the first and second modal information in the multimodal conversation information. The representation results and the corresponding conversation sequence can be input into a conversation matching model to obtain a matching result. The conversation response is then performed based on the matching result. This method can integrate multiple modal information and conversation sequence in the multimodal conversation information to respond, thereby improving the understanding of the user's previous context and thereby improving the accuracy of the response. On the other hand, by performing multimodal semantic representation on at least two modal features in the first modal information, multiple modal features in the first modal information can be explored, and the semantics of the first modal information can be fully and deeply understood, thereby improving the depth and comprehensiveness of the semantic understanding of the first modal information, thereby improving the accuracy of the response.

[0046] In step S110 , multimodal conversation information arranged in conversation time sequence is obtained.

[0047] In this example embodiment, multimodal conversation information refers to conversation information that includes at least two modal information. For example, conversation information can be information with a time sequence formed by the interactive dialogue between the user and the customer service, and the conversation time sequence refers to the time sequence in the conversation. A modality refers to the source or form of a type of information; multimodal conversation information can include at least two types of information such as image information, audio information, and text information. For example, multimodal conversation information includes first modal information and second modal information, where the first modal information is image information and the second modal information is text information or audio information. The first modal information and the second modal information are information corresponding to different modalities.

[0048] In step S120, multimodal semantic representation is performed on at least two modal features in the first modal information to obtain a first representation vector.

[0049] In this example embodiment, the modal feature refers to at least one of a text feature, an image feature, and an audio feature. Different modal features can be extracted from the first modal information by various information extraction methods. The extracted multimodal features can be semantically represented by a multimodal semantic representation model. The multimodal semantic representation model can include multiple parallel semantic representation modules and modal fusion modules. One modality can correspond to one semantic representation module. The semantic representation module is used to perform semantic representation of the corresponding modality, and the modal fusion module is used to fuse the positive results of multiple reports.

[0050] Exemplarily, as shown in FIG2 , the multimodal semantic representation model includes a first encoding model, a second encoding model, and a feature fusion model. Multimodal semantic representation can be performed by the following steps:

[0051] The first step is to extract the text content and text position information in the first modal information.

[0052] In this example embodiment, an optical character recognition model can be used to extract text content and text position information, i.e., first modal features, from the first modal information. In this example, OCR (Optical Character Recognition) can be used to extract text content and text box coordinates from image information.

[0053] The second step is to divide the first modal information into blocks to obtain block information and block position coordinates.

[0054] In this example embodiment, blocking refers to the blocking (patch) of the image, block information refers to the obtained image block, and block position coordinates refer to the position coordinates of each image block in the image. This example can extract the image block and its position coordinates, namely the first modal feature.

[0055] The third step is to vectorize the text content, text position information, block information, and block position coordinates respectively.

[0056] In this example embodiment, the text content can be segmented and converted into word vectors. For example, the words can be embedded, and the text position information and block position coordinates can also be embedded and mapped through embedding. The text position vector and the word vector are superimposed to form a vector representation of the first modal feature. The block information (image block) can be mapped into a low-dimensional vector through a linear transformation, and the block vector and the block position vector are superimposed to form a vector representation of the second modal feature. The block vector dimension is the same as the word vector dimension.

[0057] In the fourth step, the vector representation result of the block information and the vector representation result of the block position coordinates are encoded respectively through the first encoding model to obtain a first encoding vector.

[0058] In this example embodiment, the first encoding model is used to perform feature encoding on the image blocks and their position information in the first modal information. The first encoding model can be a transformer architecture that uses a multi-head attention mechanism to encode the image blocks. For example, the input of the first encoding model is the corresponding CLS token (classification feature / for global feature aggregation) and the image patch token (including the block position coordinate embedding). The block position coordinates are added to enhance the model's ability to recognize the overall layout of the page, and the output of the CLS token is used as a unimodal representation of the first modal information.

[0059] In the fifth step, the vector representation of the text content and the vector representation of the text position information are encoded separately through the second encoding model to obtain a second encoding vector.

[0060] In this example embodiment, the second encoding model is used to feature encode the text content and its position information in the first modal information. The second encoding model can be a transformer architecture that uses a multi-head attention mechanism to encode the OCR text. For example, the input of the second encoding model is the corresponding CLS token and OCR text token (including text position information embedding). By adding text position information, the model's ability to recognize text structure and overall page layout is enhanced, and the output of the CLS token is used as a unimodal representation of the second modal information.

[0061] In the sixth step, the concatenated vector of the first coding vector and the second coding vector is subjected to feature fusion through a feature fusion model to obtain a first representation vector.

[0062] In this example embodiment, the first and second encoding vectors can be concatenated and then input into a feature fusion model to fuse the feature dimensions. The feature fusion model can be a multi-layer perceptron, which performs linear transformation and nonlinear processing on the concatenated vectors to achieve deep feature fusion and output a first representation vector for image-text fusion.

[0063] The multimodal semantic representation model in the above embodiment is trained, wherein the first encoding model and the second encoding model are obtained by fine-tuning the parameters of the overall model training after a pre-training process based on contrastive learning, and the feature fusion model is obtained through overall model training.

[0064] Exemplarily, the comparative learning pre-training process for the first coding model and the second coding model may include the following steps:

[0065] A business corpus of a target business is obtained, where the business corpus includes at least two first modal corpora.

[0066] In this example implementation, the target business can be a customer service business or other business in a customer service scenario (such as pre-sales or after-sales services, financial services, etc.). The business corpus is a corpus knowledge base for the target business and can include corpora of interactions between users and customer service staff in various customer service scenarios. Corpora can include a large amount of text, images, audio, and other information. The first modality corpus can be an image corpus, and an image can serve as a first modality corpus.

[0067] The image features and text features of a first modality corpus are used as positive examples, and the image features of a first modality corpus and text features of another first modality corpus are used as negative examples to construct a first training sample.

[0068] In this example implementation, the image features and text features of the same image can be used as positive examples, and the image features of one image and the text features of another image can be used as negative examples, that is, the text features and image features of the negative examples come from different images, to construct the first training sample.

[0069] The first training sample is used to perform comparative learning training on the first coding model and the second coding model to obtain a pre-trained multimodal coding model.

[0070] In this example implementation, contrastive learning is used to train a first encoding model and a second encoding model, learning the common features between positive examples and distinguishing the differences between negative examples. The two encoding models, after contrastive learning, encode similar data similarly while minimizing the encoding differences between different data types. Contrastive learning eliminates the need to focus on the tedious details of the examples, focusing solely on distinguishing data at an abstract semantic level. This results in simpler models and optimization, and enhanced generalization capabilities.

[0071] For example, referring to FIG3 , contrastive learning training can be performed by the following steps:

[0072] The image features corresponding to the positive examples and negative examples in the first training sample are input into the first encoding model, and an image representation vector is output; the text features corresponding to the positive examples and negative examples in the first training sample are input into the second encoding model, and a text representation vector is output; the contrast loss is calculated based on the image representation vector and the text representation vector; the first encoding model and the second encoding model are trained based on the contrast loss to obtain a pre-trained multimodal encoding model.

[0073] In this example embodiment, the first encoding model is used for image representation, and the second encoding model is used for text representation. InfoNCE loss can be used for training. Specifically, the contrast loss can be expressed as: L InfoNCE =L v2l +L l2v

[0074] Among them, L v2l Corresponding to the contrastive learning loss for a given image:

[0075] L l2v Corresponding to the contrastive learning loss under a given text:

[0076] in, is the image representation vector, is the text representation vector, * can be i or j, i and j represent the image numbers respectively, τ is the temperature hyperparameter (such as set to 0.05), and the superscript T represents the transpose operation.

[0077] In this example, through unsupervised contrastive learning pre-training with a large number of positive and negative samples, the first encoding model and the second encoding model can learn two dimensions of information: first, the overall page layout of the image and the layout information of the OCR text box, which is helpful for domain judgment; second, the semantic information of the OCR text, which can be used to judge the domain from another dimension. The intention of some images can also be judged based on semantics.

[0078] Based on the above pre-training of the first encoding model and the second encoding model, the entire model can also be trained as a whole. The entire model can include a multimodal semantic representation model, a pre-trained third encoding model and a session matching model. The overall training process can include the following steps.

[0079] A retrieval knowledge base is constructed based on the business corpus. Each multimodal conversational corpus in the business corpus is used as the second training sample. Labels for the second training samples are retrieved from the retrieval knowledge base. The pre-trained first and second encoding models, pre-trained third encoding model, feature fusion model, and conversation matching model are trained using the second training samples and their corresponding labels.

[0080] In this example embodiment, since this method can be understood as a retrieval-based answering method, in order to cover all user questions and needs as much as possible, it is necessary to form a retrieval knowledge base based on each domain in the customer service scenario, user intent, and possible FAQs (Frequently Asked Questions) and answers, so that it can cover all customer service business scenarios. As shown in Figure 4, the retrieval knowledge base can include domain dimensions (domain 1, domain 2, ..., domain N), intent dimensions within domains (such as intent 1-2 within domain 1, ..., intent 1-m1) and question-answer pair dimensions under intents (such as question-answer pairs 1-1-1, ..., 1-1-p1 under intent 1-1), N is the number of domains, m1 is the number of intents within domain 1, and p1 is the number of question-answer pairs under intent 1-1. The "..." in Figure 4 indicates that specific examples are omitted here. The process of building a retrieval knowledge base can be as follows: First, customer service scenarios can be divided into multiple domains based on user entry, user source, product, function, and other dimensions. The principle of division is to ensure that there is no overlap between domains and to cover as many customer service scenarios as possible. Second, within each domain, the intent of user questions is enumerated. For example, this can be roughly divided according to the type of user questions, again ensuring that there is no overlap between intents and to cover as many user questions as possible within the domain. Finally, within a given domain and intent, a large number of FAQs and their corresponding answers are summarized to cover as many user questions as possible. Ultimately, a cascaded retrieval knowledge base of domain-intent-FAQ-answer pairs is formed. For example, for the FAQ "How to repay a loan," the retrieval knowledge base can correspond to FAQs with the repayment method question intent in the financial domain. This cascaded retrieval knowledge base can improve matching accuracy.

[0081] In this example embodiment, a conversation corpus can be used as a training sample, and a training sample can include multiple text sentences and image information. Based on the multi-dimensional retrieval knowledge base, the sample label also includes labels under each dimension, that is, one dimension corresponds to one label, and each sample can have a domain label, an intent label, and a question-answer pair label. The sample can be input into the model to fine-tune the parameters of the pre-trained model (first encoding model, second encoding model, third encoding model) and train the untrained model (feature fusion model and conversation matching model). The image information in the sample is input into the multimodal semantic representation model, and the text information in the sample is input into the third encoding model. The loss function is calculated by the output and label of the prediction head corresponding to each dimension, and the model is trained by minimizing the loss function. The loss function here can be a cross-entropy loss function. After the model training is completed, it can be used for online conversation response.

[0082] In step S130 , semantic representation of the corresponding modality is performed on the second modal information to obtain a second representation vector.

[0083] In this example implementation, the second modal information can be semantically encoded using a semantic encoding model to obtain a second representation vector. The semantic encoding model is obtained by training a pre-trained third encoding model, i.e., after the entire model is trained. The third encoding model can be a BERT model or another text semantic representation model, which is not limited in this example.

[0084] In step S140 , the first characterization vector, the second characterization vector and the corresponding session time sequence are respectively input into a session matching model to obtain a matching result.

[0085] In this example implementation, the conversation matching model may include a context representation model and multiple prediction heads. Each prediction head corresponds to a dimension in the retrieval knowledge base, i.e., there is one prediction head for each dimension, used to predict conversation information for that dimension. The context representation model may be a contextual semantic understanding model, such as a transformer model. The prediction head may be a classifier of various types, in this case a fully connected network.

[0086] Exemplarily, through the context representation model, the first representation vector, the second representation vector and the corresponding conversation sequence are feature encoded based on the multi-head attention mechanism to obtain a third representation vector; the third representation vector is input into each prediction head respectively to obtain the matching results of each dimension.

[0087] In this example embodiment, a transformer model (context representation model) can be used to perform feature encoding based on a multi-head attention mechanism on the first representation vector and the second representation vector and the corresponding conversation sequence (i.e., position encoding). The transformer model may include an encoder and a decoder. The encoder adopts a self-attention mechanism and a feedforward neural network structure, and the decoder adopts the same structure as the encoder. The difference is that the decoder also has an attention structure between the two structures, which is used to make the decoder focus on the relevant parts of the input features. The representation results output by the transformer model are respectively input into three prediction heads (i.e., domain prediction head, intent prediction head, and question-answer pair prediction head) for matching prediction in each dimension, and each prediction head outputs a corresponding matching result.

[0088] In some embodiments, each prediction head may be used to perform full connection processing and nonlinear activation processing on the third representation vector to obtain a prediction probability score corresponding to each prediction head.

[0089] In this example implementation, each prediction head can be a fully connected layer, and the characterization results are fully connected through the fully connected layer, and then mapped to the probabilities of each category through the softmax activation function. Each field or intent or question-answer pair can correspond to a category, and the fields, intents and questions involved in the conversation and the corresponding answers can be predicted. For example, the prediction results with the prediction probability scores in the top n (n is a positive integer) positions in each prediction head can be selected as the corresponding matching results. The n corresponding to different prediction heads can be the same or different, and this example does not limit this. For example, the field with the highest probability score can be determined as the field matching result, the intent with the probability scores in the top 5 can be determined as the intent matching result, and the questions and corresponding answers with the probability scores in the top 20 can be determined as the question-answer pair matching results.

[0090] In step S150 , a response is made to the multimodal conversation information according to the matching result.

[0091] In this example implementation, the matching result may be used as the response content to respond to the multimodal conversation information, or part of the content in the matching result may be selected to respond, which is not limited in this example.

[0092] In some embodiments, a keyword library can also be constructed to predict answers together with the matching results. Based on the retrieval knowledge base, a keyword library for fields and intentions can be constructed, that is, a corresponding keyword library can be constructed based on the knowledge information of the field dimension and the intention dimension of the retrieval knowledge base. It is used to perform field matching with the text in the user's question text or image to assist in predicting the field and intention. Since the information content of the keyword is relatively limited, only coarse-grained judgments such as field or intention can be made, which can be integrated with the matching results of the above-mentioned multimodal model to improve the accuracy of the answer.

[0093] Based on this, referring to Figure 5, keyword matching can be performed on both the text content and the secondary modal information in the domain keyword library; and keyword matching can be performed on both the text content and the secondary modal information in the intent keyword library. If the text content and the secondary modal information appear in either the domain keyword library or the intent keyword library, it is considered a hit, and the hit domain and intent are output.

[0094] In some embodiments, based on keyword matching, the matching results output by the model can be rearranged to improve the accuracy of the response, which can specifically include the following steps.

[0095] According to the matching results of each prediction head and the keyword matching results, the matching results of the prediction heads corresponding to the question-answer pair dimensions are rearranged to obtain a rearrangement score; the question-answer pair with a rearrangement score greater than a preset threshold is used as the response content for the conversation response.

[0096] In this example embodiment, each matched question-answer pair can be weighted according to the matching results of each prediction head and the keyword matching results to obtain a weighted re-ranking score. Question-answer pairs with a re-ranking score greater than a preset threshold are selected as the response content. The preset threshold can be set according to actual circumstances.

[0097] Exemplarily, for each matched FAQ output by the model, the re-ranking score is calculated as follows: FAQ re-ranking score = model predicted FAQ score + k1*model predicted score of the corresponding field of the FAQ + k2*model predicted score of the corresponding intent of the FAQ + k3*(flag_keyword), where k1, k2, and k3 represent the weight coefficients corresponding to the three parts respectively, and flag_keyword is a Boolean variable indicating whether the keyword is successfully matched. If so, its value is 1, otherwise it is 0.

[0098] For example, the implementation process of the conversation response method disclosed in the present invention is shown in Figure 6, which can be implemented based on a conversation response model. The conversation response model may include a multimodal representation model, a semantic encoding model, a context representation model and multiple prediction heads (domain prediction head, intent prediction head and question-answer pair prediction head). It can be specifically implemented through the following steps.

[0099] The first step is to obtain the business corpus and build a retrieval knowledge base and keyword library (domain keyword library and intent keyword library).

[0100] The second step is to construct training samples based on the business corpus and retrieval knowledge base, and use them to train the conversation response model. This mainly includes pre-training (comparative training) of the first encoding model and the second encoding model in the multimodal representation model and training of the entire conversation response model.

[0101] In the third step, the user inputs question-related content, which may include text (such as text 1, text 2, text 3) and images (image 1 and image 2). The conversation response system obtains multimodal conversation information arranged in conversation time sequence.

[0102] In the fourth step, the multimodal representation model is used to fuse the multiple modalities in the first modal information (such as image 1 and image 2) in the multimodal conversation information to obtain a first representation vector (image 1 representation and image 2 representation).

[0103] The fifth step is to semantically represent the second modal information (such as text 1, text 2, and text 3) in the multimodal conversation information through the semantic encoding model to obtain the second representation vector (text 1 representation, text 2 representation, and text 3 representation).

[0104] In the sixth step, the first representation vector, the second representation vector and the corresponding conversation sequence (conversation sequence encoding) are feature encoded based on the multi-head attention mechanism through the context representation model to obtain the third representation vector.

[0105] In the seventh step, the third representation vector is input into the domain prediction head, intent prediction head, and question-answer pair prediction head respectively to obtain the predicted domain, predicted intent, and predicted question-answer pair.

[0106] In the eighth step, in the domain keyword library, keyword matching is performed on the text content in the first modal information and the second modal information respectively; in the intent keyword library, keyword matching is performed on the text content in the first modal information and the second modal information respectively.

[0107] In the ninth step, according to the matching results of each prediction head and the keyword matching results, the matching results of the prediction head corresponding to the question-answer dimension are rearranged to obtain the rearrangement score.

[0108] In the tenth step, question-answer pairs with rearrangement scores greater than a preset threshold are used as response content for conversational response.

[0109] During the conversation response process in the above embodiment, the training of the entire model is mainly for models that have not been pre-trained, such as the feature fusion model, the context representation model and multiple prediction heads, and other pre-trained models only perform parameter fine-tuning.

[0110] Existing intelligent customer service technology is mainly based on natural language processing technology, which can better understand text-based user input (such as input text or input voice converted into text) and output corresponding answers. However, with the development of intelligent customer service technology and the diversification of user needs, in most intelligent customer service scenarios, users can more conveniently input pictures, supplemented by text to more accurately describe problems and needs. In other words, the user input modality has changed from a single text modality to multimodal information of text / audio and image, which poses a huge challenge to intelligent customer service technology based on natural language processing. Related technologies are often unable to correctly understand and judge when encountering image information. Ignoring the pictures in the conversation will result in an inability to correctly and completely understand the user's previous context and intentions, and it is easy to get answers that do not conform to the current context and user needs, resulting in reduced response accuracy and user experience.

[0111] In response to the problems faced by users' multimodal input (text + pictures), this paper designs a retrieval-based intelligent customer service response method based on a multimodal model of images and texts. It fuses multiple modal information in image information through a multimodal representation model, and at the same time fuses the position coordinate information of images and texts, thereby enhancing the model's ability to recognize text structure and overall page layout. Combining a semantic encoding model with a contextual representation model, it can simultaneously process information in the text and image modalities input by users, and through multiple prediction heads, accurately understand the context and user intent, that is, the intent / domain to which the user's current question belongs, and within this intent / domain, screen out answers that meet the current context and user needs, which can significantly improve the accuracy of responses and user experience, and better cope with diversified intelligent customer service scenarios.

[0112] By building a cascaded search knowledge base, the present invention can understand and match the conversation semantics from coarse to fine level, thereby improving the accuracy of model prediction. Based on model prediction, the present invention combines keyword library matching to increase the accuracy of the granularity of the response results to the question answer.

[0113] Furthermore, in this example embodiment, a conversation response device 700 is also provided. The conversation response device 700 can be applied to a server. Referring to FIG7 , the conversation response device 700 may include: an acquisition module 710, a first representation module 720, a second representation module 730, a conversation matching module 740, and a response module 750, wherein: the acquisition module 710 is configured to acquire multimodal conversation information arranged in conversation time sequence, the multimodal conversation information including first modal information and second modal information; the first representation module 720 is configured to perform multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector; the second representation module 730 is configured to perform semantic representation of the corresponding modality on the second modal information to obtain a second representation vector; the conversation matching module 740 is configured to input the first representation vector and the second representation vector and the corresponding conversation time sequence into a conversation matching model to obtain a matching result; and the response module 750 is configured to respond to the multimodal conversation information based on the matching result.

[0114] In an exemplary embodiment of the present disclosure, the first representation module 720 includes: an extraction submodule, configured to extract text content and text position information from the first modal information; a blocking submodule, configured to block the first modal information to obtain block information and block position coordinates; a vectorization submodule, configured to vectorize the text content, text position information, block information, and block position coordinates respectively; and a semantic representation submodule, configured to perform multimodal semantic representation on the vectorized representation result through a multimodal semantic representation model to obtain a first representation vector.

[0115] In an exemplary embodiment of the present disclosure, the vectorization submodule is further configured to: perform embedding mapping on text content, text position information, and block position coordinates respectively; and perform linear mapping on block information.

[0116] In an exemplary embodiment of the present disclosure, the multimodal semantic representation model includes a first encoding model, a second encoding model and a feature fusion model, and the semantic representation submodule is further configured to: encode the vector representation result of the block information and the vector representation result of the block position coordinates respectively through the first encoding model to obtain a first encoding vector; encode the vector representation of the text content and the vector representation of the text position information respectively through the second encoding model to obtain a second encoding vector; and perform feature fusion on the concatenated vector of the first encoding vector and the second encoding vector through the feature fusion model to obtain a first representation vector; wherein the first encoding model and the second encoding model are obtained through a pre-training process based on contrastive learning.

[0117] In an exemplary embodiment of the present disclosure, the device 700 also includes a first training module, which is configured to: obtain a business corpus of the target business, the business corpus including at least two first modal corpora; use the image features and text features of one first modal corpus as positive examples, and use the image features of one first modal corpus and the text features of another first modal corpus as negative examples to construct a first training sample; use the first training sample to perform comparative learning training on the first encoding model and the second encoding model to obtain a pre-trained multimodal encoding model.

[0118] In an exemplary embodiment of the present disclosure, the first training module is further configured to: input the image features corresponding to the positive examples and negative examples in the first training sample into the first encoding model, and output the image representation vector; input the text features corresponding to the positive examples and negative examples in the first training sample into the second encoding model, and output the text representation vector; calculate the contrast loss based on the image representation vector and the text representation vector; and train the first encoding model and the second encoding model based on the contrast loss to obtain a pre-trained multimodal encoding model.

[0119] In an exemplary embodiment of the present disclosure, the second representation module 730 is further configured to: semantically encode the second modal information through a semantic coding model to obtain a second representation vector, where the semantic coding model is obtained by training a pre-trained third coding model.

[0120] In an exemplary embodiment of the present disclosure, the device 700 also includes a second training module, and the second training module is further configured to: use each multimodal conversation corpus in the business corpus as a second training sample; retrieve the label of the second training sample from the retrieval knowledge base, the retrieval knowledge base is constructed based on the business corpus, and the retrieval knowledge base includes domain dimensions, intention dimensions within the domain, and question-answer pair dimensions under the intention, and the labels include labels under each dimension; use the second training samples and the corresponding labels to train the pre-trained first encoding model, second encoding model, third encoding model, feature fusion model, and conversation matching model respectively.

[0121] In an exemplary embodiment of the present disclosure, the conversation matching model includes a context representation model and multiple prediction heads, one prediction head corresponds to a dimension in the retrieval knowledge base, and the conversation matching module 740 is further configured to: perform feature encoding based on a multi-head attention mechanism on the first representation vector and the second representation vector and the corresponding conversation sequence through the context representation model to obtain a third representation vector; and input the third representation vector into each prediction head respectively to obtain matching results corresponding to each dimension.

[0122] In an exemplary embodiment of the present disclosure, the session matching module 740 is further configured to: use each prediction head to perform full connection processing and nonlinear activation processing on the third representation vector to obtain a prediction probability score corresponding to each prediction head; and determine the prediction result with the prediction probability score in the first n digits in each prediction head as the matching result, where n is a positive integer.

[0123] In an exemplary embodiment of the present disclosure, the device 700 further includes a keyword matching module, which is configured to: perform keyword matching on the text content and the second modal information in the domain keyword library; and perform keyword matching on the text content and the second modal information in the intent keyword library; wherein the domain keyword library and the intent keyword library are respectively constructed based on the knowledge information of the corresponding dimensions of the retrieval knowledge base. The response module 750 is further configured to: re-arrange the matching results of the prediction heads corresponding to the dimensions of the question-answer pair based on the matching results of each prediction head and the keyword matching results to obtain a re-arrangement score; and use the question-answer pair with a re-arrangement score greater than a preset threshold as the response content for the conversation response.

[0124] The specific details of each module or unit in the above-mentioned conversation response device have been described in detail in the corresponding conversation response method, so they will not be repeated here.

[0125] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device may implement the steps shown in Figures 1 to 3 and Figures 5 to 6.

[0126] It should be noted that the computer-readable medium described in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0127] The electronic device 800 according to this embodiment of the present disclosure is described below with reference to Figure 8. The electronic device 800 shown in Figure 8 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] As shown in FIG8 , electronic device 800 is implemented as a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, the aforementioned at least one processing unit 810, the aforementioned at least one storage unit 820, a bus 830 connecting various system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0129] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0130] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0131] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0132] Bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0133] The electronic device 800 may also communicate with one or more external devices 870 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RA identification systems, tape drives, and data backup storage systems.

[0134] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0135] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0136] It should be noted that although the steps of the method of the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additional or alternative steps, such as omitting certain steps, combining multiple steps into one step, and / or decomposing a step into multiple steps, should all be considered part of this disclosure.

[0137] It should be understood that the present disclosure disclosed and defined in this specification extends to all alternative combinations of two or more individual features mentioned or evident in the text and / or drawings. All of these different combinations constitute multiple alternative aspects of the present disclosure. The embodiments of this specification illustrate the best mode known for implementing the present disclosure and will enable those skilled in the art to utilize the present disclosure.

Claims

1. A conversation answering method, comprising: Acquire multimodal conversation information arranged in a conversation time sequence, wherein the multimodal conversation information includes first modal information and second modal information; Performing multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector; Performing semantic representation of the corresponding modality on the second modal information to obtain a second representation vector; Inputting the first representation vector, the second representation vector and the corresponding session time sequence into a session matching model respectively to obtain a matching result; The multimodal conversation information is responded to according to the matching result.

2. The conversation answering method according to claim 1, wherein: The performing multimodal semantic representation on at least two modal features in the first modal information includes: Extracting text content and text position information from the first modal information; Divide the first modal information into blocks to obtain block information and block position coordinates; Respectively vectorizing the text content, the text position information, the block information, and the block position coordinates; The vectorized representation result is subjected to multimodal semantic representation through a multimodal semantic representation model to obtain the first representation vector.

3. The conversation answering method according to claim 2, wherein: The vectorizing the text content, the text position information, the block information, and the block position coordinates respectively includes: Embedding and mapping the text content, the text position information and the block position coordinates respectively; The block information is linearly mapped.

4. The conversation answering method according to claim 2 or 3, wherein: The multimodal semantic representation model includes a first encoding model, a second encoding model and a feature fusion model, and the multimodal semantic representation of the vectorized representation result includes: By using a first coding model, encoding the vector representation result of the block information and the vector representation result of the block position coordinates are respectively performed to obtain a first coding vector; By using a second coding model, encoding the vector representation of the text content and the vector representation of the text position information are respectively performed to obtain a second coding vector; Performing feature fusion on the concatenated vector of the first encoding vector and the second encoding vector through a feature fusion model to obtain the first representation vector; The first coding model and the second coding model are obtained through a pre-training process based on contrastive learning.

5. The conversation answering method according to claim 4, wherein: The method further comprises: Acquire a business corpus of a target business, wherein the business corpus includes at least two first modal corpora; The image features and text features of the first modal corpus are taken as positive examples, and the image features and text features of the first modal corpus are taken as positive examples. The image feature and another text feature of the first modal corpus are used as negative examples to construct a first training sample; The first training sample is used to perform comparative learning training on the first coding model and the second coding model to obtain a pre-trained multimodal coding model.

6. The conversation answering method according to claim 5, wherein: The using the first training sample to perform comparative learning training on the first coding model and the second coding model includes: Inputting image features corresponding to positive examples and negative examples in the first training sample into a first encoding model, and outputting an image representation vector; inputting text features corresponding to positive examples and negative examples in the first training sample into a second encoding model, and outputting a text representation vector; Calculating contrast loss based on the image representation vector and the text representation vector; The first coding model and the second coding model are trained according to the contrast loss to obtain a pre-trained multimodal coding model.

7. The conversation answering method according to claim 1, wherein: The performing semantic representation of the corresponding modality on the second modal information includes: The second modal information is semantically encoded through a semantic encoding model to obtain the second representation vector, and the semantic encoding model is obtained by training a pre-trained third encoding model.

8. The conversation answering method according to claim 5, wherein: The method further comprises: Using each multimodal conversation corpus in the business corpus as a second training sample; Retrieving a label of the second training sample in a retrieval knowledge base, wherein the retrieval knowledge base is constructed based on the business corpus, the retrieval knowledge base includes a domain dimension, an intent dimension within the domain, and a question-answer pair dimension under the intent, and the label includes a label under each dimension; The pre-trained first encoding model, the second encoding model, the third encoding model, the feature fusion model, and the session matching model are trained respectively using the second training samples and the corresponding labels.

9. The conversation answering method according to claim 8, wherein: The session matching model includes a context representation model and a plurality of prediction heads, one prediction head corresponds to a dimension in the retrieval knowledge base, and obtaining the matching result includes: By using the context representation model, feature encoding is performed on the first representation vector, the second representation vector and the corresponding session sequence based on a multi-head attention mechanism to obtain a third representation vector; The third representation vector is input into each of the prediction heads respectively to obtain matching results corresponding to each dimension.

10. The conversation answering method according to claim 9, wherein: The obtaining of matching results in each dimension includes: Using each of the prediction heads, performing full connection processing and nonlinear activation processing on the third representation vector to obtain a prediction probability score corresponding to each prediction head; Determine the prediction result whose prediction probability score is in the first n places in each prediction head as the matching result, where n is a positive integer.

11. The conversation answering method according to claim 9 or 10, wherein: The method further comprises: In the domain keyword library, keyword matching is performed on the text content and the second modal information respectively; In the intention keyword library, keyword matching is performed on the text content and the second modal information respectively; Wherein, the domain keyword library and the intention keyword library are respectively constructed based on the knowledge information of the corresponding dimensions of the retrieval knowledge base; The step of responding to the multimodal conversation information according to the matching result includes: According to the matching results of each prediction head and the keyword matching results, the matching results of the prediction heads corresponding to the question-answer pair dimensions are rearranged to obtain a rearrangement score; The question-answer pair whose rearrangement score is greater than a preset threshold is used as the response content for conversation response.

12. A conversation answering device, comprising: An acquisition module is configured to acquire multimodal conversation information arranged in a conversation time sequence, wherein the multimodal conversation information includes first modal information and second modal information; a first representation module configured to perform multimodal semantic representation on at least two modal features in the first modal information to obtain a first representation vector; A second representation module is configured to perform semantic representation of the second modal information in a corresponding modality to obtain a second representation vector; a session matching module, configured to input the first representation vector, the second representation vector and the corresponding session time sequence into a session matching model respectively to obtain a matching result; The response module is configured to respond to the multimodal conversation information according to the matching result.

13. An electronic device, comprising: one or more processors; as well as A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images