Question answering method and device, electronic device, and storage medium

Abstract

The present disclosure provides a method and device for answering questions, an electronic device, and a storage medium; and relates to the field of information processing technology. The method includes: obtaining first modal features and second modal features corresponding to each first modal information in multimodal information; for a first modal information, performing semantic similarity matching on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, to obtain a first similarity vector representation and a second similarity vector representation; performing information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; in response to a multimodal similarity score being greater than or equal to a preset threshold, answering the question using the corresponding first modal information as the answer content. The present disclosure can solve the problem of low answer accuracy in the related art due to ignoring other modal information in the document.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to a question answering method and device, an electronic device, and a storage medium. Background Art

[0002] Document Question Answering (DocQA) is a very important technology that can extract answers from unstructured document information. Currently, most document question answering technologies are based on deep neural networks to perform semantic matching on document content to determine the answer content, ignoring other modal information in the document content, resulting in low answer accuracy.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a question answering method and device, an electronic device, and a storage medium, thereby solving to a certain extent the problem of low answer accuracy caused by ignoring other modal information in the document in the related art.

[0005] According to a first aspect of the present disclosure, a question answering method is provided, comprising: obtaining first modal features and second modal features corresponding to each first modal information in multimodal information; for a first modal information, performing semantic similarity matching on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly; performing information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; in response to a multimodal similarity score being greater than or equal to a preset threshold, answering the question using the corresponding first modal information as the answer content.

[0006] Optionally, semantic similarity matching is performed on the first modal feature and the user question feature, including: performing semantic similarity matching on the first modal feature and the user question feature through multiple attention layers; and performing nonlinear activation and linear transformation on the output of the multiple attention layers through a feedforward network layer to obtain a first similarity vector representation.

[0007] Optionally, semantic similarity matching is performed on the first modal feature and the user question feature through multiple attention layers, including: for each attention layer, performing a first linear processing on the user question feature to obtain a first intermediate vector; performing a second linear processing and a third linear processing on the first modal feature, respectively, to obtain a second intermediate vector and a third intermediate vector; determining the similarity between the first intermediate vector and the second intermediate vector; performing weighted matching on the third intermediate vector based on the similarity to obtain the output of the current attention layer; performing vector splicing on the outputs of each attention layer, and performing full connection processing on the spliced ​​vector through the first fully connected layer to obtain the output of the multiple attention layers.

[0008] Optionally, semantic similarity matching is performed on the second modal feature and the user question feature, including: concatenating the second modal feature and the user question feature to obtain a concatenated vector; and mapping the concatenated vector to a low-dimensional space through a second fully connected layer to obtain a second similarity vector representation.

[0009] Optionally, information fusion is performed on the first similarity vector representation and the second similarity vector representation, including: information fusion is performed on the first similarity vector representation and the second similarity vector representation through a gating network; and full connection processing and nonlinear activation processing are performed on the fused vector to obtain a multimodal similarity score.

[0010] Optionally, the first modal information includes image information, and the second modal features include text features, and obtaining the second modal features corresponding to each first modal information includes: extracting title information of an image information; identifying the content and attribute information in the image information through a pre-trained target detection model; extracting candidate text information in the image information through optical character recognition; vectorizing the title information, content and attribute information, and candidate text information to obtain text features corresponding to the image information.

[0011] Optionally, obtaining first modal features corresponding to each first modal information includes: extracting image vector information from each first modal information through an image pre-training model to obtain corresponding first modal features.

[0012] Optionally, the method further includes: in response to receiving a user question sentence, encoding the user question sentence through a pre-trained encoding model to obtain a user question feature.

[0013] Optionally, the multimodal information includes second modal information, and the method further includes: in response to each multimodal similarity score being less than a preset threshold, determining the response content based on the second modal information.

[0014] Optionally, the response content is determined based on the second modal information, including: splitting the second modal information into paragraphs and determining keywords for each paragraph obtained by the splitting; aggregating semantically similar paragraphs in each paragraph based on the keywords; condensing the aggregated paragraphs through a pre-trained summary generation model to obtain summary text; and using the summary text as the response content.

[0015] Optionally, the method further includes: in response to a plurality of multimodal similarity scores being greater than or equal to a preset threshold, determining that the first modal information corresponding to the highest multimodal similarity score is the response content.

[0016] According to a second aspect of the present disclosure, there is provided a question answering device, the device comprising: an acquisition module, a semantic matching module, a fusion module and a response module, wherein: the acquisition module is configured to acquire first modal features and second modal features corresponding to each first modal information in multimodal information; the semantic matching module is configured to perform semantic similarity matching on the first modal features and user question features, and on the second modal features and user question features, for a first modal information, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly; the fusion module is configured to perform information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; the response module is configured to respond to a multimodal similarity score being greater than or equal to a preset threshold value, using the corresponding first modal information as the response content to answer the question.

[0017] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

[0018] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to execute the method of any of the above embodiments.

[0019] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0020] In the question answering method provided in the example implementation mode of the present disclosure, on the one hand, by performing semantic similarity matching on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, and fusing the two semantic similarity matching results, multiple modal features in one modal information can be mined, and the multiple modal features can be fused to determine the answer content, thereby improving the accuracy of the answer. On the other hand, when the multimodal similarity score is greater than or equal to a preset threshold, the original first modal information can be used as the answer content, avoiding the deep semantic extraction process of the document content, achieving a rapid response of the answering system, and improving the answering efficiency. In addition, the first modal features and the second modal features of each first modal information can be pre-extracted to further improve the answering efficiency.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0023] Figure 1 The flowchart of the question answering method according to one embodiment of the present disclosure is schematically shown.

[0024] Figure 2 The entity diagram of the second modality feature extraction process in the first modality information according to one embodiment of the present disclosure is schematically shown.

[0025] Figure 3 An image corresponding to the first modality information in an embodiment according to the present disclosure is schematically shown.

[0026] Figure 4 The following is a schematic diagram of a semantic similarity matching process between a first modality feature and a user question feature according to an embodiment of the present disclosure.

[0027] Figure 5 The flowchart of the process of determining a multimodal similarity score according to one embodiment of the present disclosure is schematically shown.

[0028] Figure 6 A flowchart schematically shows a processing procedure of a question answering method according to an embodiment of the present disclosure.

[0029] Figure 7The structure block diagram of the question answering device according to one embodiment of the present disclosure is schematically shown.

[0030] Figure 8 A block diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] The example of the present invention provides an exemplary system diagram of an application scenario of a question answering method and device, and the system includes a terminal and a server. The method of this embodiment can be applied to a server. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, or a node in a blockchain.

[0034] For example, during the customer service process, the user sends a user question statement on the user client, and the server can associate multimodal information based on the user question statement, and obtain the first modal features and second modal features corresponding to each first modal information in the multimodal information from the knowledge base; for a first modal information, the first modal feature and the user question feature, and the second modal feature and the user question feature, are semantically matched, and a first similarity vector representation and a second similarity vector representation are correspondingly obtained; information fusion is performed on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; in response to a multimodal similarity score being greater than or equal to a preset threshold, the corresponding first modal information is determined to be the response content, and the response content is sent to the user terminal through the client.

[0035] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted device, etc., but is not limited thereto. When the question answering method provided in this embodiment is implemented through the interaction between the terminal and the server, the terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present disclosure does not limit this.

[0036] The question answering method provided in the embodiment of the present disclosure may be executed on a server, and accordingly, the question answering device is generally disposed in the server.

[0037] The following describes the question answering method disclosed in the embodiments of this specification in conjunction with specific embodiments.

[0038] refer to Figure 1 As shown, a question answering method according to an exemplary embodiment of the present disclosure can be applied to various question answering systems and may include the following steps.

[0039] Step S110, obtaining first modal features and second modal features corresponding to each first modal information in the multimodal information.

[0040] Step S120, for a first modal information, semantic similarity matching is performed on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly.

[0041] Step S130: performing information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score.

[0042] Step S140, in response to a multimodal similarity score being greater than or equal to a preset threshold, determining that the corresponding first modal information is the response content.

[0043] In the question answering method provided in this example embodiment, on the one hand, by performing semantic similarity matching on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, and fusing the two semantic similarity matching results, multiple modal features in one modal information can be mined, and the multiple modal features can be fused to determine the answer content, thereby improving the accuracy of the answer. On the other hand, when the multimodal similarity score is greater than or equal to a preset threshold, the original first modal information can be used as the answer content, avoiding the deep semantic extraction process of the document content, achieving a rapid response of the answering system, and improving the answering efficiency. In addition, the first modal features and the second modal features of each first modal information can be pre-extracted to further improve the answering efficiency.

[0044] In step S110, first modal features and second modal features corresponding to each first modal information in the multimodal information are obtained.

[0045] In this example implementation, multimodal information refers to document information including at least two modal information. A modality refers to a source or form of information; multimodal information may include at least two of image information, audio information, and text information. First modal information refers to a modal information including at least two modal features in multimodal information, such as image information or audio information. The first modal feature and the second modal feature are two features corresponding to different modalities.

[0046] Exemplarily, for a web page search scenario, all information corresponding to each web page can be considered as multimodal information. The image information and text information included in the web page are different modal information. An image information can be regarded as a first modal information, an image feature in an image information can be regarded as a first modal feature, and a text feature in the image information can be regarded as a second modal feature.

[0047] In this example implementation, the multimodal information can be pre-processed to extract the first modal features and the second modal features, and the first modal features and the second modal features can be stored in a knowledge base so that they can be directly called during online responses to reduce response time. The first modal features and the second modal features can be extracted through various pre-trained models.

[0048] Exemplarily, the image pre-training model can be used to extract image vector information from each first modal information to obtain the first modal features corresponding to each first modal information.

[0049] In this example implementation, the image pre-training model is used to identify each first modal information (image information), and the image pre-training model can be various pre-trained convolutional neural networks, such as CNN, VGG16, VGG19, ResNet-50, etc., which are not limited in this example. Exemplarily, feature extraction is performed through multiple convolutional layers and pooling layers of VGG16, and then the output result, i.e., the first modal feature, is obtained through multiple fully connected layers and softmax activation functions.

[0050] Exemplarily, the first modality information includes image information, and the second modality features include text features, such as Figure 2 As shown, the second modal features corresponding to each first modal information can be obtained through the following steps.

[0051] Extract header information of an image information.

[0052] In this example implementation, when the image information has a title, it is sufficient to extract the label of the image information; when the image information does not have a title, an image title generation model can be used to convert the image-level information into title information. The image title generation model can be an existing image title generation model or tool, such as the visual processing deep learning library LAVIS, which is an open source library for training and evaluating state-of-the-art language-vision models on a rich series of common tasks and datasets, and for performing off-the-shelf reasoning on customized language-vision data, for example, inputting an image and outputting a corresponding text description.

[0053] The pre-trained target detection model is used to identify the content and attribute information in the image information.

[0054] In this example implementation, the target detection model is used to find all the target objects of interest in the image and determine the category and location of the object. The target detection model can be one of R-CNN, SPP-Net, Fast R-CNN, Faster R-CNN, and R-FCN, etc., which is not limited in this example. The content in the image information can include various objects (such as environment, people, objects, etc.), and the content attribute information can include basic attribute information such as object name and gender, and can also include attribute information such as object occupation, object color, object height, etc., which is not limited in this example.

[0055] Through optical character recognition, candidate text information is extracted from the image information.

[0056] In this example implementation, the text in the image information may be extracted by using optical character recognition (OCR) technology, that is, the text information contained in the image itself (that is, the candidate text information) may be extracted.

[0057] The title information, content and attribute information, and candidate text information are vectorized to obtain text features corresponding to the image information.

[0058] In this example implementation, the title information, content and attribute information, and candidate text information extracted in the above steps can be used as text information extracted from the image information (i.e., the second modal information extracted from the first modal information), and the text features are vectorized to obtain the corresponding second modal features. A pre-trained text vectorization model can be used for vectorization, for example, a pre-trained Bert model can be used.

[0059] For example, Figure 3 As shown in the figure, the title information extracted from the image is "Convenient parking lot", the extracted content and its attribute information include "white car", "blue sign", "woods", "building", etc., and the extracted candidate text information is "Parking lot exit; Warm reminder, please park your vehicle in an orderly manner in the monitoring coverage area...". The above extracted information is vectorized (using the pre-trained Bert model) to obtain the second modal features of the image (i.e., the text features of the image).

[0060] Exemplarily, the first modal features and the second modal features extracted by the above embodiment can be stored in two knowledge bases respectively. The content parsing process of the first modal features and the second modal features can be obtained by pre-processing the image information in the target document and stored in the knowledge base, so as to improve the subsequent response speed.

[0061] The present invention is directed to a multimodal document question-answering system. In some embodiments, the method further includes: in response to receiving a user question sentence, encoding the user question sentence through a pre-trained encoding model to obtain user question features.

[0062] In this example implementation, the user question statement may be a question statement sent by a user in a question-and-answer system, such as a user question sentence received by a customer service system. The received user question statement may be vectorized and the pre-trained encoding model may be a vectorized representation model that is the same as the second modal feature, such as a pre-trained Bert model.

[0063] In step S120, for a first modal information, semantic similarity matching is performed on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly.

[0064] In this example implementation, the first modal information may be an image, and semantic similarity matching may be performed on each image in the target document and the user question feature. The first modal feature may correspond to an image feature vector, and the second feature vector may correspond to a text feature vector extracted from the image, and the two modal features may be semantically matched with the user question feature. For example, a feature vector of the matching relationship between the two feature vectors to be matched may be modeled based on the two feature vectors to be matched, and then other models (such as a multi-layer perceptron MLP) may be used to learn a general text relationship function mapping.

[0065] Exemplarily, the semantic similarity matching of the first modal feature vector and the user question feature can be performed through a multimodal transformer model. The transformer model may include an encoder and a decoder. The encoder adopts a self-attention mechanism and a feedforward neural network structure, and the decoder adopts the same structure as the encoder. The difference is that the decoder also has an attention structure between the two structures, which is used to make the decoder focus on the relevant part of the input feature, that is, the output of the self-attention mechanism in the decoder and the output of the encoder are used to calculate the attention score, and then enter the feedforward neural network of the decoder. The encoder may include multiple encoding modules of the above structure, and the decoder may also include multiple identical decoding modules. Specifically, the following steps may be included.

[0066] Through the multi-layer attention layer, the semantic similarity matching of the first modal feature and the user question feature is performed. Through the feedforward network layer, the output of the multi-layer attention layer is nonlinearly activated and linearly transformed to obtain the first similarity vector representation.

[0067] In this example implementation, a multi-layer attention layer is used for matching based on a self-attention mechanism, and the user question feature can be used to map to Q, and the first modal feature (image vector) is used as the K and V vectors for processing based on the self-attention mechanism. The feedforward network layer is used to perform nonlinear activation and linear transformation on the output of the multi-layer attention layer, such as performing multiple linear transformations and ReLU activation function processing. The input of the encoder and decoder also includes the position information of the vector. The output of the feedforward neural network layer can be a context vector based on the self-attention mechanism, and the vector can be fully connected to obtain the first similarity vector representation (the similarity representation vector of the two input vectors).

[0068] For example, Figure 4 As shown, the semantic similarity matching between the first modal features and the user question features can be performed through multiple attention layers.

[0069] For each attention layer, the first linear layer is used to perform the first linear processing on the user question feature to obtain the first intermediate vector; the second linear layer and the third linear layer are used to perform the second linear processing and the third linear processing on the first modal feature respectively to obtain the second intermediate vector and the third intermediate vector; the similarity between the first intermediate vector and the second intermediate vector is determined; according to the similarity, the third intermediate vector is weighted matched to obtain the output of the current attention layer; the outputs of each attention layer are vector spliced, and the spliced ​​vectors are fully connected through the first fully connected layer to obtain the output of multiple attention layers.

[0070] In this example implementation, the first intermediate vector may correspond to Q (Query), the second intermediate vector may correspond to K (Key), and the third intermediate vector may correspond to V (Value). The similarity between the two may be determined by calculating the dot product between Q and K; the similarity may also be scaled using a scaled attention layer. The result is then normalized to a probability distribution using a Softmax operation, and then multiplied by V as a weighted weight for weighted summation to obtain the output of the current attention layer. Each attention layer performs the same processing, and the model parameters of different layers may be different, to obtain vector representations of multiple self-attention mechanisms. The outputs of multiple attention layers are vectorized and then fully connected using a fully connected layer (the first fully connected layer) to obtain the output of multiple attention layers.

[0071] For example, the processing of an attention layer can be expressed as follows:

[0072]

[0073] Q=W q Q i +b q

[0074] K=W k I n +b k

[0075] V=W v I n +b v

[0076] In the formula, Attention(Q, K, V) represents the output of the attention layer, softmax represents the softmax function, Q, K, and V represent the first intermediate vector, the second intermediate vector, and the third intermediate vector respectively; d represents the dimension of the K vector, which is used here Scaling is used to prevent the results from being too large; W q 、b q represents the model parameters of the first linear process, W k 、bk represents the model parameters of the second linear process, W v 、b v represents the model parameters of the third linear process, Q i Indicates the user's problem characteristics, I n represents the first modal feature, and the superscript T represents the matrix transpose operation.

[0077] In some embodiments, various existing semantic similarity matching models may be used to perform semantic similarity matching on the second modality features and the user question features.

[0078] Exemplarily, the second modal feature and the user question feature may be concatenated to obtain a concatenated vector; the concatenated vector may be mapped to a low-dimensional space through a second fully connected layer to obtain a second similarity vector representation.

[0079] In this example implementation, the second modality feature may be a text feature extracted from image information, and a vector may be mapped to a low-dimensional space through a fully connected layer.

[0080] In step S130, information fusion is performed on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score.

[0081] In this example implementation, information fusion may be performed on the first similarity vector representation and the second similarity vector representation through a gating mechanism.

[0082] Exemplarily, the first similarity vector representation and the second similarity vector representation can be fused through a gating network; the fused vectors are fully connected and nonlinearly activated to obtain a multimodal similarity score.

[0083] In this example implementation, the gated network can be a simple gated mechanism (such as a threshold hyperparameter g of 0.3) or a learned gated network model, which is not limited in this example. For example, the information fusion process can be expressed as:

[0084] g=δ(W i V iq +W t V tq +b)

[0085]

[0086] y=softmax(FNN(V m ))

[0087] In the formula, represents the dot product operation, V m represents the output of the gating network, V iq represents the first similarity vector representation, Vtq represents the second similarity vector representation, W i , W t , b represents the model parameters of the gating network, δ represents the normalized activation function, FNN represents the fully connected processing, and y represents the output multimodal similarity score.

[0088] In the above embodiment, if Figure 5 As shown, the process of determining the multimodal similarity score can use a multimodal matching model, and the multimodal matching model can be trained based on different application fields using training samples of the application field, such as customer service field, web search field, etc. The model may include a first similarity matching module, a second similarity matching module, a gating network, a fully connected layer, and an output layer. The first similarity matching module is a transformer-based matching model, and the second similarity matching model may include a linear mapping layer. The user question feature and the first modality feature are input into the first similarity matching model, and the user question feature and the second modality feature are input into the second similarity matching model. The output layer outputs the multimodal similarity score between the question and the image.

[0089] In step S140, in response to a multimodal similarity score being greater than or equal to a preset threshold, the corresponding first modal information is used as the response content to answer the question.

[0090] In this example implementation, the preset threshold is the judgment condition for the first modal response. The threshold can be set based on experience and actual conditions (such as being set to 0.9), or it can be adjusted according to different situations. This example does not limit this. When the multimodal similarity score between an image and a user's question is greater than or equal to the threshold, the first modal information (i.e., the image) can be used as the response statement to increase the information richness and diversity of the response content. For example, when the user's question statement is "the appearance or appearance of xx object", using the first modal information (including image information) to respond can give users a more intuitive answer and improve user satisfaction.

[0091] Exemplarily, when there are multiple multimodal similarity scores greater than or equal to a preset threshold, the first modal information corresponding to the highest multimodal similarity score is determined as the answer content. For example, if the multimodal similarity scores of multiple images and the user's question meet the matching condition (greater than or equal to the preset threshold), the image with the highest similarity score can be used as the answer content to answer the question.

[0092] In some embodiments, the multimodal information includes second modal information, and the method further includes: in response to each multimodal similarity score being less than a preset threshold, determining the response content based on the second modal information.

[0093] In this example implementation, the second modality information may be text-based information, and when the image matching threshold is not reached, a matching response may be made based on the text-based information.

[0094] Exemplarily, the second modal information can be segmented into paragraphs, and the keywords of each segmented paragraph can be determined; based on the keywords, the semantically similar paragraphs in each paragraph can be aggregated; through the pre-trained summary generation model, the aggregated paragraphs can be condensed to obtain summary text; and the summary text can be used as the response content.

[0095] In this example implementation, the second modal information refers to document information other than the first modal information in the multimodal information. For example, for a document containing text and images, the second modal information may be text information other than the image. Paragraph splitting can be performed using a paragraph splitting tool, and keyword extraction can be performed using an existing keyword extraction algorithm, such as a TF-IDF (Term-Frequency Inverse Document Frequency) algorithm, a TextRank algorithm, or a combination of the two. Keywords may also be titles of each paragraph. When the previous and subsequent paragraphs / titles share a specified number of keywords, the corresponding paragraphs are aggregated into a large paragraph so that the same paragraph describes the same content. A pre-trained summary generation model (such as a CNN-based Seqence-to-Sequence model, a Long Short-Term Memory Network (LSTM)-based Seqence-to-Sequence model, or an attention-based Seqence-to-Sequence model) may be used to abbreviate the text into semantically concise short sentences for each aggregated long paragraph. For example, a long paragraph of 100 sentences may be condensed into 10 sentences, or the sentences may be paraphrased, which is not limited in this example.

[0096] In the method disclosed herein, the extraction process of the first modal features and the second modal features of the first modal information in the multimodal information and the processing process of the second modal information can be completed in advance, and the extraction / processing results can be stored in the corresponding knowledge base, and the online question and answer system can obtain the corresponding information from the knowledge base for a response.

[0097] For example, the implementation process of the question answering method disclosed in the present invention is as follows: Figure 6 As shown, it can be implemented by the following steps.

[0098] Step S601: The user sends a user question statement to the answering system.

[0099] Step S602: The answering system encodes the user's question statement through a pre-trained encoding model to obtain user question features.

[0100] Step S603: The answering system determines the multimodal information corresponding to the target document based on the received user question statement.

[0101] In this example, the relevant document content can be associated with the user's question statement as the target document. If the target document has multimodal information, the process can proceed to S604. If the target document has multimodal information, the target document can be condensed and answered.

[0102] In step S604, the response system obtains the first modal features and the second modal features corresponding to each first modal information in the multimodal information from the knowledge base.

[0103] Step S605: condense the second modal information in the multimodal information through the pre-trained summary generation model to obtain a summary text.

[0104] Step S606: The answering system performs semantic similarity matching between the first modal feature and the user question feature for a first modal information through a first similarity matching model.

[0105] In this example, the first similarity matching model is a transformer model, in which user question features are used to map to Q, and the first modal features are used as K and V information.

[0106] Step S607: The answering system performs semantic similarity matching on the second modal feature and the user question feature through a second similarity matching model.

[0107] In this example, the second similarity matching model is a fully connected network.

[0108] In step S608, the response system performs information fusion on the first similarity vector representation and the second similarity vector representation through a gated network to obtain a multimodal similarity score.

[0109] In step S609, the response system determines whether there is at least one multimodal similarity score greater than or equal to a preset threshold, and if so, proceeds to step S610, otherwise proceeds to step S611.

[0110] Step S610: The response system sends the first modal information with the highest multimodal similarity score to the user as the response content.

[0111] Step S611, the response system sends the summary text in step S605 to the user as the response content.

[0112] In the question answering process in the above embodiments, models related to semantic similarity matching, such as the first similarity matching model, the second similarity matching model, and the gating network, need to be trained, and other models can be used directly using pre-trained models.

[0113] The present disclosure aims at the situation that most documents in real applications have multimodal information expressions such as text content, picture content, and audio content. On the one hand, a document content parsing method based on multimodality is proposed to extract text information from various aspects of the first modal information containing image content, fuse the visual information and text information in the image, and improve the matching accuracy of the image. At the same time, based on the current situation of uploading document content in advance, the multimodal information in the document can be pre-parsed to improve the system response speed during the answering process. On the other hand, by semantically matching the two dimensions of the image content, namely the first modal feature and the user question feature, and the semantic similarity matching of the second modal feature and the user question feature, the matching accuracy is improved, thereby improving the answer accuracy. In addition, in the case of a high similarity score of the image content, directly using the image for answering can improve the answering efficiency and answering effect, and enhance the user experience.

[0114] Furthermore, in this exemplary embodiment, a question answering device 800 is also provided. The question answering device 700 can be applied to a server. Figure 7 As shown, the question answering device 700 may include: an acquisition module 710, a semantic matching module 720, a fusion module 730 and a determination module 740, wherein the acquisition module 710 is configured to acquire the first modal features and the second modal features corresponding to each first modal information in the multimodal information; the semantic matching module 720 is configured to perform semantic similarity matching on the first modal features and the user question features, and on the second modal features and the user question features, for a first modal information, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly; the fusion module 730 is configured to perform information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; the answering module 740 is configured to answer the question using the corresponding first modal information as the answer content in response to a multimodal similarity score being greater than or equal to a preset threshold.

[0115] In an exemplary embodiment of the present disclosure, the semantic matching module 720 includes a first matching submodule, which is configured to: perform semantic similarity matching on the first modal feature and the user question feature through a multi-layer attention layer; and perform nonlinear activation and linear transformation on the output of the multi-layer attention layer through a feedforward network layer to obtain a first similarity vector representation.

[0116] In an exemplary embodiment of the present disclosure, the first matching submodule is further configured to: for each attention layer, perform a first linear processing on the user question feature to obtain a first intermediate vector; perform a second linear processing and a third linear processing on the first modal feature, respectively, to obtain a second intermediate vector and a third intermediate vector; determine the similarity between the first intermediate vector and the second intermediate vector; perform weighted matching on the third intermediate vector based on the similarity to obtain the output of the current attention layer; perform vector splicing on the outputs of each attention layer, and perform full connection processing on the spliced ​​vector through the first fully connected layer to obtain the output of multiple attention layers.

[0117] In an exemplary embodiment of the present disclosure, the semantic matching module 720 also includes a second matching submodule, which is configured to: concatenate the second modal feature and the user question feature to obtain a concatenated vector; and map the concatenated vector to a low-dimensional space through a second fully connected layer to obtain a second similarity vector representation.

[0118] In an exemplary embodiment of the present disclosure, the fusion module 730 is further configured to: perform information fusion on the first similarity vector representation and the second similarity vector representation through a gating network; perform full connection processing and nonlinear activation processing on the fused vector to obtain a multimodal similarity score.

[0119] In an exemplary embodiment of the present disclosure, the first modal information includes image information, the second modal feature includes text feature, and the acquisition module 710 includes a first extraction submodule, which is configured to: extract title information of an image information; identify the content and attribute information in the image information through a pre-trained target detection model; extract candidate text information in the image information through optical character recognition; vectorize the title information, content and attribute information, and candidate text information to obtain text features corresponding to the image information.

[0120] In an exemplary embodiment of the present disclosure, the acquisition module 710 also includes a second extraction submodule, and the second extraction submodule is configured to: extract image vector information from each first modality information through an image pre-training model to obtain corresponding first modality features.

[0121] In an exemplary embodiment of the present disclosure, the apparatus 700 further includes an encoding module, which is configured to, in response to receiving a user question sentence, encode the user question sentence through a pre-trained encoding model to obtain a user question feature.

[0122] In an exemplary embodiment of the present disclosure, the multimodal information includes second modal information, and the response module 740 is further configured to: in response to each multimodal similarity score being less than a preset threshold, determine the response content based on the second modal information.

[0123] In an exemplary embodiment of the present disclosure, the response module 740 is further configured to: split the second modal information into paragraphs and determine the keywords of each paragraph obtained by the splitting; based on the keywords, aggregate the semantically similar paragraphs in each paragraph; condense the sentences of the aggregated paragraphs through a pre-trained summary generation model to obtain a summary text; and use the summary text as the response content.

[0124] In an exemplary embodiment of the present disclosure, the response module 740 is further configured to: in response to multiple multimodal similarity scores being greater than or equal to a preset threshold, determine that the first modal information corresponding to the highest multimodal similarity score is the response content.

[0125] The specific details of each module or unit in the above-mentioned question answering device have been described in detail in the corresponding question answering method, so they will not be repeated here.

[0126] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method in the following embodiment. For example, the electronic device may implement Figure 1 , Figure 2 and Figure 4-Figure 6 The various steps shown, etc.

[0127] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0128] Refer to the following Figure 8 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0130] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0131] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0132] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0133] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0134] The electronic device 800 may also communicate with one or more external devices 870 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA identification systems, tape drives, and data backup storage systems, etc.

[0135] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0136] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0137] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in the specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc., which shall all be considered as part of the present disclosure.

[0138] It should be understood that the disclosure disclosed and defined in this specification extends to all alternative combinations of two or more individual features mentioned or evident in the text and / or the accompanying drawings. All these different combinations constitute multiple alternative aspects of the disclosure. The embodiments of this specification illustrate the best mode known for implementing the disclosure and will enable those skilled in the art to utilize the disclosure.

Claims

1. A question answering method, characterized in that: include: Acquire first modal features and second modal features corresponding to each first modal information in the multimodal information; For the first modal information, semantic similarity matching is performed on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, to obtain a first similarity vector representation and a second similarity vector representation correspondingly; Performing information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; In response to the multimodal similarity score being greater than or equal to a preset threshold, the corresponding first modal information is used as the response content to answer the question.

2. The question answering method according to claim 1, characterized in that: The performing semantic similarity matching on the first modal feature and the user question feature includes: Performing semantic similarity matching between the first modal feature and the user question feature through multiple attention layers; Through the feedforward network layer, the output of the multi-layer attention layer is nonlinearly activated and linearly transformed to obtain the first similarity vector representation.

3. The question answering method according to claim 2, characterized in that: The performing semantic similarity matching on the first modal feature and the user question feature through the multi-layer attention layer includes: For each attention layer, performing a first linear processing on the user question feature to obtain a first intermediate vector; Performing a second linear processing and a third linear processing on the first modal feature respectively to obtain a second intermediate vector and a third intermediate vector; determining a similarity between the first intermediate vector and the second intermediate vector; According to the similarity, weighted matching is performed on the third intermediate vector to obtain an output of the current attention layer; The outputs of each attention layer are vectorized and concatenated, and the concatenated vectors are fully connected through the first fully connected layer to obtain the outputs of the multi-layer attention layer.

4. The question answering method according to claim 1, characterized in that: The performing semantic similarity matching on the second modality feature and the user question feature includes: Concatenating the second modal feature and the user question feature to obtain a concatenated vector; The concatenated vector is mapped to a low-dimensional space through a second fully connected layer to obtain the second similarity vector representation.

5. The question answering method according to claim 1, characterized in that: The fusing information of the first similarity vector representation and the second similarity vector representation includes: Performing information fusion on the first similarity vector representation and the second similarity vector representation through a gating network; The fused vector is subjected to full connection processing and nonlinear activation processing to obtain the multimodal similarity score.

6. The question answering method according to claim 1, characterized in that: The first modal information includes image information, the second modal features include text features, and the acquiring the second modal features corresponding to each first modal information includes: Extracting title information of the image information; Identify the content and attribute information of the image information through a pre-trained object detection model; Extracting candidate text information from the image information through optical character recognition; The title information, the content and its attribute information, and the candidate text information are vectorized to obtain text features corresponding to the image information.

7. The question answering method according to claim 1, characterized in that: The obtaining of the first modal features corresponding to each first modal information includes: Through the image pre-training model, the image vector information in each of the first modal information is extracted to obtain the corresponding first modal features.

8. The question answering method according to claim 6, characterized in that: The method further comprises: In response to receiving a user question sentence, the user question sentence is encoded by a pre-trained encoding model to obtain the user question feature.

9. The question answering method according to any one of claims 1 to 8, characterized in that: The multimodal information includes second modal information, and the method further includes: In response to each of the multimodal similarity scores being less than a preset threshold, determining a response content based on the second modal information.

10. The question answering method according to claim 9, characterized in that: The determining the response content based on the second modal information includes: Splitting the second modal information into paragraphs, and determining keywords of each paragraph obtained by the splitting; Based on the keywords, the paragraphs with similar semantics are aggregated in each paragraph; The pre-trained summary generation model is used to condense the sentences of the aggregated paragraphs to obtain a summary text; and the summary text is used as the response content.

11. The question answering method according to claim 1, characterized in that: The method further comprises: In response to a plurality of the multimodal similarity scores being greater than or equal to a preset threshold, determining the first modal information corresponding to the highest multimodal similarity score as the response content.

12. A question answering device, characterized in that: The device comprises: An acquisition module is configured to acquire first modal features and second modal features corresponding to each first modal information in the multimodal information; A semantic matching module is configured to perform semantic similarity matching on the first modal feature and the user question feature, and on the second modal feature and the user question feature, respectively, for the first modal information, and obtain a first similarity vector representation and a second similarity vector representation accordingly; a fusion module, configured to perform information fusion on the first similarity vector representation and the second similarity vector representation to obtain a multimodal similarity score; The answering module is configured to answer the question using the corresponding first modal information as the answering content in response to the multimodal similarity score being greater than or equal to a preset threshold.

13. An electronic device, characterized in that: include: one or more processors; as well as A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images