Multi-modal question and answer method and device, electronic equipment and computer readable storage medium

By acquiring and processing images, tables, and text information, and combining pre-trained models to perform multimodal Q&A, the problem that existing intelligent Q&A systems cannot handle complex multimodal problems, achieving a better user experience.

CN119047583APending Publication Date: 2024-11-29PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411195133.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing intelligent question and answer system is mainly based on text types and cannot effectively deal with complex multimodal problems, resulting in poor user experience.

Method used

By obtaining image information, table information and initial text information, feature extraction and conversion processing are performed, combined with pre-trained image feature extraction model, text feature extraction model and relationship generation model, relationship category text is generated, and the self-attention mechanism model is used for Multimodal Q&A.

Benefits of technology

It breaks through the limitations of single-modal question-and-answer questions, realizes effective answers to multimodal questions, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047583A_ABST
    Figure CN119047583A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of financial science and technology, and provides a multi-modal question and answer method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the feature extraction of image information, obtaining image text information and image description information, and carrying out the conversion processing of table information, and obtaining table text information; inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text; splicing the context information and the obtained user question to obtain question associated information; and inputting the question associated information into a pre-trained self-attention mechanism model to obtain a question answering result. According to the technical scheme, the limitation of question answering is broken through, and better use experience is brought to the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of financial technology, and in particular to a multimodal question-answering method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of social economy and the continuous advancement of science and technology, people's living standards are constantly improving. In the financial industry, intelligent question-and-answer systems have been increasingly used in people's daily lives. However, current intelligent question-and-answer systems generally provide single-modal answers based on text types, that is, given a piece of text content, they look for answers from the text content based on the user's questions. However, this question-and-answer method has limitations and cannot answer more complex modal types. The questions that can be answered are also limited, which brings a bad user experience to users. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide a multimodal question-answering method, device, electronic device and computer-readable storage medium, which break through the limitations of the questions answered and bring users a better user experience.

[0005] In a first aspect, an embodiment of the present application provides a multimodal question-answering method, comprising:

[0006] Get image information, table information and initial text information;

[0007] Extracting features from the image information to obtain image text information and image description information, and converting the table information to obtain table text information;

[0008] Inputting the image information into a pre-trained image feature extraction model to obtain an image feature vector, and inputting the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector;

[0009] Inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text;

[0010] Determine context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text;

[0011] The context information and the acquired user question are combined to obtain question-related information;

[0012] The question-related information is input into the pre-trained self-attention mechanism model to obtain the question answering result.

[0013] According to some embodiments of the present application, extracting features from the image information to obtain image text information and image description information includes:

[0014] Inputting the image information into a pre-trained optical character recognition model to obtain the image text information;

[0015] Inputting the image information into a pre-trained bilingual visual language model to perform image feature encoding to obtain image encoding information;

[0016] The image encoding information is input into the text decoder of the pre-trained multi-head attention mechanism model to obtain the image description information.

[0017] According to some embodiments of the present application, inputting the picture encoding information into a text decoder of a pre-trained multi-head attention mechanism model to obtain the image description information includes:

[0018] Determine multiple candidate words according to the image encoding information and the multi-head attention mechanism model;

[0019] When the number of the candidate words is greater than a preset sampling value, taking the plurality of candidate words as target candidate words;

[0020] In a case where the candidate probability corresponding to the target candidate word is greater than a preset sampling probability, the image description information is determined according to the target candidate word.

[0021] According to some embodiments of the present application, inputting the image information into a pre-trained optical character recognition model to obtain the image text information includes:

[0022] Performing character recognition processing on the image information according to the optical character recognition model to obtain a plurality of initial recognition characters;

[0023] The plurality of initially recognized characters are matched with a preset recognition database to obtain the image text information.

[0024] According to some embodiments of the present application, converting the table information to obtain table text information includes:

[0025] Determine the number of rows and columns, table header name and table position information from the table information;

[0026] The table information is formatted according to the number of rows and columns, the table header name and the table position information to obtain the table text information.

[0027] According to some embodiments of the present application, after inputting the question-related information into a pre-trained self-attention mechanism model to obtain a question answering result, the method further includes:

[0028] Determine a question answer result identifier according to the user question;

[0029] Determine the answer loss value according to the question answering result and the question answering result identifier;

[0030] The self-attention mechanism model is subjected to parameter adjustment processing according to the answer loss value.

[0031] According to some embodiments of the present application, determining the context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text includes:

[0032] Determine image feature information according to the image text information and the image description information;

[0033] The image feature information, the table text information, the initial text information and the relationship category text are combined and processed to obtain the context information.

[0034] In a second aspect, an embodiment of the present application further provides a multimodal question-answering device, the device comprising:

[0035] A first processing module, used for acquiring image information, table information and initial text information;

[0036] A second processing module is used to extract features from the image information to obtain image text information and image description information, and to convert the table information to obtain table text information;

[0037] A third processing module is used to input the image information into a pre-trained image feature extraction model to obtain an image feature vector, and to input the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector;

[0038] A fourth processing module, used for inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text;

[0039] a fifth processing module, configured to determine context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text;

[0040] A sixth processing module, configured to combine the context information with the acquired user question to obtain question-related information;

[0041] The seventh processing module is used to input the question-related information into a pre-trained self-attention mechanism model to obtain a question-answering result.

[0042] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal question and answer method as described in the first aspect above is implemented.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the multimodal question-answering method described in the first aspect above.

[0044] The multimodal question-answering method according to the embodiment provided by the present application has at least the following beneficial effects: in the process of multimodal question-answering, firstly, image information, table information and initial text information are obtained, then feature extraction is performed on the image information to obtain image text information and image description information, and conversion processing is performed on the table information to obtain table text information; then the image information is input into a pre-trained image feature extraction model to obtain an image feature vector; then the table text information and the initial text information are input into a pre-trained text feature extraction model to obtain a text feature vector; then the image feature vector and the text feature vector are input into a pre-trained relationship generation model to obtain a relationship category text; then the context information can be determined based on the image text information, image description information, table text information, initial text information and relationship category text; then the context information and the obtained user question are spliced ​​to obtain question-related information; finally, the question-related information is input into a pre-trained self-attention mechanism model to obtain the question answering result. Through the above technical solution, three different modalities of data information, namely image information, table information and initial text information, are integrated to facilitate multi-modal answers to questions asked by users, breaking through the limitations of the questions answered and bringing users a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0046] Figure 1is a flowchart of a multimodal question-answering method provided by an embodiment of the present application;

[0047] Figure 2 is a specific flow chart of step S200 provided in one embodiment of the present application;

[0048] Figure 3 is a specific flow chart of step S230 provided by an embodiment of the present application;

[0049] Figure 4 is a specific flow chart of step S210 provided in one embodiment of the present application;

[0050] Figure 5 is a specific flow chart of step S200 provided in another embodiment of the present application;

[0051] Figure 6 is a flowchart of a multimodal question-answering method provided by another embodiment of the present application;

[0052] Figure 7 is a specific flow chart of step S500 provided in one embodiment of the present application;

[0053] Figure 8 is a schematic diagram of a multimodal question-answering device provided by an embodiment of the present application;

[0054] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0057] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0061] Artificial intelligence is AI. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0063] The present application provides a multimodal question-answering method, device, electronic device and computer-readable storage medium. In the process of multimodal question-answering, image information, table information and initial text information are first obtained, and then feature extraction is performed on the image information to obtain image text information and image description information, and conversion processing is performed on the table information to obtain table text information; then the image information is input into a pre-trained image feature extraction model to obtain an image feature vector; then the table text information and the initial text information are input into a pre-trained text feature extraction model to obtain a text feature vector; then the image feature vector and the text feature vector are input into a pre-trained relationship generation model to obtain a relationship category text; then the context information can be determined based on the image text information, image description information, table text information, initial text information and relationship category text; then the context information and the obtained user question are spliced ​​to obtain question-related information; finally, the question-related information is input into a pre-trained self-attention mechanism model to obtain the question answering result. Through the above technical solution, three different modalities of data information, namely image information, table information and initial text information, are integrated to facilitate multi-modal answers to questions asked by users, breaking through the limitations of the questions answered and bringing users a better user experience.

[0064] The multimodal question-answering method provided in the embodiment of the present application relates to the field of financial technology. The multimodal question-answering method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the multimodal question-answering method, etc., but is not limited to the above forms.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0067] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, Figure 1 It is a flowchart of a multimodal question-answering method provided by an embodiment of the present application. The multimodal question-answering method includes but is not limited to steps S100 to S700.

[0069] Step S100, acquiring image information, table information and initial text information;

[0070] Step S200, extracting features from the image information to obtain image text information and image description information, and converting the table information to obtain table text information;

[0071] Step S300, inputting the image information into a pre-trained image feature extraction model to obtain an image feature vector, and inputting the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector;

[0072] Step S400, inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text;

[0073] Step S500, determining context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text;

[0074] Step S600, concatenating the context information and the acquired user question to obtain question-related information;

[0075] Step S700, inputting the question-related information into a pre-trained self-attention mechanism model to obtain a question answering result.

[0076] It should be noted that in the process of multimodal question answering, first obtain image information, table information and initial text information, then extract features from the image information to obtain image text information and image description information, and convert the table information to obtain table text information; then input the image information into the pre-trained image feature extraction model to obtain the image feature vector; then input the table text information and the initial text information into the pre-trained text feature extraction model to obtain the text feature vector; then input the image feature vector and the text feature vector into the pre-trained relationship generation model to obtain the relationship category text; then determine the context information based on the image text information, image description information, table text information, initial text information and relationship category text; then concatenate the context information and the obtained user question to obtain the question association information; finally input the question association information into the pre-trained self-attention mechanism model to obtain the question answering result. Through the above technical solution, the data information of three different modes of image information, table information and initial text information are integrated to facilitate multimodal answers to the questions asked by users, break through the limitations of the answered questions, and bring users a better user experience.

[0077] It should be noted that the image information in the embodiments of the present application may be images of the financial industry, for example, images introducing insurance types, images introducing the entire insurance process, and images introducing the insurance claims process. The tabular information may include a table introducing different types of insurance. For example, one column of the table is different types of insurance, and another column of the table is the insurance introduction content corresponding to the different types of insurance. The initial text information may include text content introducing different investment strategies, and may also include text content explaining different fund dividends, which is not limited here.

[0078] It is worth noting that image text information and image description information can be obtained by extracting features from image information; image text information is the text description information in the image, for example, if there is text introducing different types of insurance in an image, then the text content on the image is the image text information; image description information is the description of the objects in the image, for example, if there is a tree and a person in an image, then the image description information of the image can include information describing the "tree" and the "person". Tabular text information can be obtained by converting the tabular information. Through the above technical solution, the prerequisite is prepared for the subsequent fusion of image text information, image description information, tabular information and initial text information.

[0079] It is worth noting that by inputting the image information into the pre-trained image feature extraction model, the image feature vector can be obtained, wherein the pre-trained image feature extraction model is an image feature extraction model that has been trained. Based on the image feature extraction model, the feature vector of the image information can be extracted, thereby obtaining the image feature vector. And by inputting the tabular text information and the initial text information into the pre-trained text feature extraction model, the text feature vector can be obtained, wherein the pre-trained text feature extraction model is an image feature extraction model that has been trained. Based on the tabular text information and the initial text information, the text feature extraction model can be used to extract text features, thereby obtaining the text feature vector. By inputting the image feature vector and the text feature vector into the pre-trained relationship generation model, the relationship category text can be obtained. The image feature vector and the text feature vector can be fused to realize the fusion of different modal data, so as to prepare for the subsequent multi-modal question and answer.

[0080] It is worth noting that after obtaining the relationship category text, the context information can be determined based on the image text information, image description information, table text information, initial text information and relationship category text, so as to prepare for the subsequent multimodal question and answer. Then, the context information and the obtained user questions are concatenated to obtain the question-related information, and the obtained question-related information is input into the pre-trained self-attention mechanism model to obtain the corresponding question answering result; through the above method, multimodal question answering can be well realized, so that question answers will no longer be restricted to text answers, bringing users a better user experience.

[0081] It should be noted that the image feature extraction model is a key technology in the field of computer vision. It is used to extract information from images that helps understand the content of the image. These features can be visual elements such as color, texture, shape, and edge. They are essential for tasks such as image recognition, classification, detection, and segmentation. Text feature extraction models are algorithms and techniques used in natural language processing to extract useful information from text data. These models can convert raw text into a numerical form that can be processed by machine learning algorithms, namely feature vectors. The relational generative model is an artificial intelligence technology that combines the capabilities of generative models and relational reasoning to generate data or make predictions. This model is particularly suitable for tasks that require understanding the relationships between entities, such as social network analysis, recommendation systems, and knowledge graph construction. The core idea of ​​the relational generative model is to learn patterns and relationships in the data and then use this knowledge to generate new data instances or make relational predictions. The self-attention mechanism model, also known as the internal attention mechanism, is a deep learning technique that allows the model to focus on different parts of the input sequence when processing data and to capture dependencies within the sequence; this mechanism is very important in fields such as natural language processing and computer vision, especially when processing sequence data; the core idea of ​​the self-attention mechanism is to calculate the attention weight of each element in the sequence for all other elements, and then aggregate information based on these weights.

[0082] In some embodiments, Figure 2 As shown, the above step S200 may include but is not limited to steps S210 to S230.

[0083] Step S210, inputting the image information into a pre-trained optical character recognition model to obtain image text information;

[0084] Step S220, inputting the image information into a pre-trained bilingual visual language model to perform image feature encoding to obtain image encoding information;

[0085] Step S230, input the picture encoding information into the text decoder of the pre-trained multi-head attention mechanism model to obtain image description information.

[0086] It should be noted that in the process of extracting features from image information to obtain image text information and image description information, the image information is first input into a pre-trained optical character recognition model to obtain the image text information; then the image information is input into a pre-trained bilingual visual language model for image feature encoding to obtain image encoding information; finally, the image encoding information is input into a text decoder of a pre-trained multi-head attention mechanism model to obtain image description information; through the above technical solution, the prerequisite is prepared for subsequent multimodal data fusion.

[0087] It is worth noting that the optical character recognition model is a technology used to convert text in various types of documents into machine-encoded text; this conversion makes the text in the image that was originally uneditable and unsearchable become editable and searchable text data. For example, an image contains insurance introduction content, so the optical character recognition model can be used to identify the insurance introduction content in the image to determine the insurance introduction content information in the image.

[0088] It is worth noting that a bilingual visual-language model is an artificial intelligence model that can simultaneously process and understand visual information (such as images) and two languages ​​(usually Chinese and English); this model combines the techniques of computer vision and natural language processing and can perform tasks such as image description, visual question answering, and cross-language text-to-image conversion in a multilingual environment.

[0089] It is worth noting that the multi-head attention mechanism model is an advanced attention model in deep learning. It is an extension of the traditional attention mechanism, enabling the model to simultaneously focus on multiple different parts or aspects of the input sequence. This mechanism is particularly useful in natural language processing tasks such as text translation, text summarization, question answering, etc.

[0090] In some embodiments, Figure 3 As shown, the above step S230 may also include but is not limited to steps S231 to S233.

[0091] Step S231, determining multiple candidate words according to the image encoding information and the multi-head attention mechanism model;

[0092] Step S232, when the number of candidate words is greater than a preset sampling value, taking the multiple candidate words as target candidate words;

[0093] Step S233, when the candidate probability corresponding to the target candidate word is greater than the preset sampling probability, determine the image description information according to the target candidate word.

[0094] It should be noted that, in the process of inputting the picture coding information into the text decoder of the pre-trained multi-head attention mechanism model to obtain the image description information, first, multiple candidate words are determined according to the picture coding information and the multi-head attention mechanism model; then, when the number of candidate words is greater than the preset sampling value, the multiple candidate words can be used as target candidate words; when the candidate probability corresponding to the target candidate word is greater than the preset sampling probability, the image description information can be determined based on the target candidate word; through the above technical solution, the image description information can be generated more accurately based on the text decoder of the multi-head attention mechanism model.

[0095] It is worth noting that both the sampling value and the sampling probability can be set according to actual conditions. By setting different sampling values ​​and sampling probabilities, the image description information can be better set.

[0096] In some embodiments, Figure 4 As shown, the above step S210 may also include but is not limited to steps S211 to S212.

[0097] Step S211, performing character recognition processing on the image information according to the optical character recognition model to obtain a plurality of initial recognition characters;

[0098] Step S212, matching the plurality of initially recognized characters with a preset recognition database to obtain image text information.

[0099] It should be noted that, in the process of inputting image information into a pre-trained optical character recognition model to obtain image text information, the image information is first subjected to character recognition processing according to the optical character recognition model to obtain a plurality of initial recognition characters; then, the obtained plurality of initial recognition characters can be subsequently matched with a preset recognition database to obtain the corresponding image text information; through the above technical solution, more accurate character recognition processing can be performed on the image information according to the optical character recognition model to better obtain the image text information.

[0100] Exemplarily, exemplary, a picture includes fund introduction content, so the fund introduction content on the picture can be identified and processed through the optical character recognition model to determine the fund introduction content information on the picture.

[0101] It is worth noting that the recognition database is a database that pre-stores recognition words, and the database contains multiple different recognition words. By matching and comparing multiple initial recognition words with the recognition words in the recognition database, the corresponding image text information can be obtained more accurately.

[0102] In some embodiments, Figure 5 As shown, the above step S200 may include but is not limited to steps S240 to S250.

[0103] Step S240, determining the number of rows and columns, the table header name and the table position information from the table information;

[0104] Step S250, performing format conversion processing on the table information according to the number of rows and columns, the table header name and the table position information to obtain table text information.

[0105] It should be noted that in the process of converting the table information to obtain the table text information, the number of rows and columns, the table header name and the table position information are first determined from the table information; then the table information can be formatted according to the number of rows and columns, the table header name and the table position information to obtain the table text information. Through the above technical solution, the table information is formatted to obtain the corresponding table text information, which is a prerequisite for the subsequent determination of the text feature vector.

[0106] For example, for a table containing insurance introduction content, the number of rows and columns, the header name and the table location information in the table can be determined first; then the table information can be formatted according to the number of rows and columns, the header name and the table location information to obtain the table text information of the table.

[0107] In some embodiments, Figure 6 As shown, after executing step S700, steps S810 to S830 may also be included but not limited to.

[0108] Step S810, determining a question answer result identifier according to the user question;

[0109] Step S820, determining the answer loss value according to the question answering result and the question answering result identifier;

[0110] Step S830, adjusting the parameters of the self-attention mechanism model according to the answer loss value.

[0111] It should be noted that after the question-related information is input into the pre-trained self-attention mechanism model and the question answering result is obtained, the question answering result identifier can be determined based on the user question; then the answer loss value can be determined based on the question answering result and the question answering result identifier; finally, the self-attention mechanism model can be parameter adjusted based on the answer loss value so that multimodal question answering processing can be performed more accurately in the subsequent process.

[0112] It is worth noting that the answer loss value can be determined based on the question answering result and the question answering result identifier; subsequently, the parameters of the self-attention mechanism model can be adjusted based on the answer loss value, so that the self-attention mechanism model can subsequently answer multimodal questions more accurately.

[0113] In some embodiments, Figure 7 As shown, the above step S500 may include but is not limited to step S510 to step S520.

[0114] Step S510, determining image feature information according to image text information and image description information;

[0115] Step S520 , combining the image feature information, the table text information, the initial text information and the relationship category text to obtain context information.

[0116] It should be noted that, in the process of determining context information based on image text information, image description information, table text information, initial text information and relationship category text, the image feature information is first determined based on the image text information and the image description information; then, the image feature information, table text information, initial text information and relationship category text can be combined and processed to obtain the context information; question-related information can be obtained by splicing the context information and the acquired user questions, in order to prepare for multimodal answer processing.

[0117] In order to more clearly explain the multimodal question-answering method mentioned in the embodiments of the present application, a specific embodiment is used for illustration below.

[0118] The embodiment of the present application defines C={I, T, P} as given context information, where I is an image, T is a table, and p is text.

[0119] For I, two types of information need to be extracted: one is the optical character recognition information in the image, and the other is the description information of the image:

[0120] Iocr is the image text information extracted using the EasyOCR model, and Icap is the text description information of the image. The image description uses the BLIP model. First, input the image into the BLIP model for image feature encoding, and use the encoded image features as the input of the ViT model text decoder. The ViT model text decoder will output the generated text, which is the information description of the image. In the BLIP generation process, this method uses an elite candidate strategy that combines top-k and top-p to perform beam search for text words. The beam search method uses a conditional probability model, that is, the generation of the current text will depend on the previous first word to the previous word:

[0121]

[0122] The sampling strategy of top-k is to manually set the value of k first, and then no matter what the probability of the currently generated word is, as long as the number of generated candidate words is less than k, it will be sampled as a candidate word:

[0123]

[0124] The sampling strategy of top-p is to set the probability p. No matter how many candidate words are currently generated, as long as they are greater than this p value, they will be sampled as candidate words:

[0125]

[0126] The advantage of this is that it can ensure that no matter how low the probability of the word generated at the current moment is, it can sample the candidate word. At the same time, when the number of generated candidate words is guaranteed, the one with the highest probability can be sampled. Finally, the candidate words generated at each moment are input into the next moment, and the description of the picture generated by the ViT model Fcaption() is concatenated with |ocr to obtain | text :

[0127] I text =F caption (I, K, p);

[0128] Convert table structured data into text format:

[0129] T text = "header:h 1 |…|h N

[0130] row 1:r11|…|r1N

[0131] …

[0132] row M:rM1|…|rMN";

[0133] Where M and N are the number of rows and columns in the table, header is the name of the column in the table, and r11 represents the first row and first column. text In text form, C = {I text , T text ,P}, maps all information to a textual representation.

[0134] Input the pure image into a pre-trained image feature extraction model CL|P to extract the image feature vector:

[0135] I feature =F v (I);

[0136] Then, the table is converted into text format and the input text is input into a pre-trained text feature extraction model BERT to obtain text features:

[0137] P feature =F l (T text , P);

[0138] Will I feature and P featureInput a Retionale generation model together (the input is the image features and text features, and the output is the relationship category between the two, such as no relationship, inclusion relationship, and included relationship), and obtain the fused R vector:

[0139] R=F r (I feature , P feature );

[0140] Then the new context information can be expressed as:

[0141] C text = {I text , T text , P, R};

[0142] In C text middle, I text It is the fusion of the text information after OCR in the image and the text information of the picture description. text is pure table text information, P is pure text information, and R is the relational category text derived from the features of the image and text. These four types of text are concatenated with the user's question Q to obtain the final text type input X.

[0143] Take X as the input of a Transformer decoder Fqa:

[0144] Y=F qa (X);

[0145] The final output Y is the result of the problem.

[0146] When the model is trained, the loss value of Y output by Fqa and the manually labeled result y will be calculated:

[0147]

[0148] Among them, L is the number of words in the answer corresponding to a sample input during training, is the manually annotated answer, yl is the answer generated by the model, and i is the i-th word in the l-th answer, that is, the input of the log function is to calculate how many words generated by the model are the same as the words in the manually annotated answer. The NLL loss value is the error between the answer generated by the calculation model and the manually annotated answer. The model will use this error to train the parameters in all models (BERT, BLIP, CLIP, ViT, Transformer, Retionale) through gradient backpropagation (EasyOCR will not be trained, it is only used). The trained model can answer the user's input question Q when used. The input when used is Q and C text The concatenated text.

[0149] Through the above technical solution, the embodiment of the present application designs a question-answering model based on three fuzzy fusions of pictures, tables, and texts, converts picture information and table information into text descriptions, and uses the text information after text splicing to generate answers to users' questions, avoiding the problem of non-convergence and instability of model training in multimodality. At the same time, it provides convenience for generating multi-target and multi-type content answers.

[0150] In addition, if Figure 8 As shown, an embodiment of the present application further provides a multimodal question-answering device 10, comprising:

[0151] The first processing module 100 is used to obtain image information, table information and initial text information;

[0152] The second processing module 200 is used to extract features from the image information to obtain image text information and image description information, and to convert the table information to obtain table text information;

[0153] The third processing module 300 is used to input the image information into a pre-trained image feature extraction model to obtain an image feature vector, and to input the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector;

[0154] The fourth processing module 400 is used to input the image feature vector and the text feature vector into the pre-trained relationship generation model to obtain the relationship category text;

[0155] A fifth processing module 500, for determining context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text;

[0156] The sixth processing module 600 is used to combine the context information and the acquired user question to obtain question-related information;

[0157] The seventh processing module 700 is used to input the question-related information into the pre-trained self-attention mechanism model to obtain the question answering result.

[0158] It should be noted that in the process of multimodal question answering, first obtain image information, table information and initial text information, then extract features from the image information to obtain image text information and image description information, and convert the table information to obtain table text information; then input the image information into the pre-trained image feature extraction model to obtain the image feature vector; then input the table text information and the initial text information into the pre-trained text feature extraction model to obtain the text feature vector; then input the image feature vector and the text feature vector into the pre-trained relationship generation model to obtain the relationship category text; then determine the context information based on the image text information, image description information, table text information, initial text information and relationship category text; then concatenate the context information and the obtained user question to obtain the question association information; finally input the question association information into the pre-trained self-attention mechanism model to obtain the question answering result. Through the above technical solution, the data information of three different modes of image information, table information and initial text information are integrated to facilitate multimodal answers to the questions asked by users, break through the limitations of the answered questions, and bring users a better user experience.

[0159] The specific implementation of the multimodal question-answering device 10 is substantially the same as the specific implementation of the multimodal question-answering method described above, and will not be described in detail herein.

[0160] In addition, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 800 , which includes: a memory 820 , a processor 810 , and a computer program stored in the memory 820 and executable on the processor 810 .

[0161] The processor 810 and the memory 820 may be connected via a bus or other means.

[0162] The non-transient software programs and instructions required to implement the multimodal question and answer method of the above-mentioned embodiments are stored in the memory 820. When executed by the processor 810, the multimodal question and answer method of the above-mentioned embodiments is executed, for example, the multimodal question and answer method as described above is executed.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor 810 or a controller, for example, by a processor 810 in the above-mentioned device embodiment, so that the above-mentioned processor 810 can execute the multimodal question-answering method in the above-mentioned embodiment.

[0165] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0166] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0167] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0168] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0169] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0170] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0171] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0172] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0174] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0178] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0179] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (Transitory Med ia), such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0181] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0182] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0183] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multimodal question answering method, characterized in that: The method comprises: Get image information, table information and initial text information; Extracting features from the image information to obtain image text information and image description information, and converting the table information to obtain table text information; Inputting the image information into a pre-trained image feature extraction model to obtain an image feature vector, and inputting the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector; Inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text; Determine context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text; The context information and the acquired user question are combined to obtain question-related information; The question-related information is input into the pre-trained self-attention mechanism model to obtain the question answering result.

2. The multimodal question answering method according to claim 1, characterized in that: The step of extracting features from the image information to obtain image text information and image description information includes: Inputting the image information into a pre-trained optical character recognition model to obtain the image text information; Inputting the image information into a pre-trained bilingual visual language model to perform image feature encoding to obtain image encoding information; The image encoding information is input into the text decoder of the pre-trained multi-head attention mechanism model to obtain the image description information.

3. The multimodal question answering method according to claim 2, characterized in that: The step of inputting the picture encoding information into a text decoder of a pre-trained multi-head attention mechanism model to obtain the image description information includes: Determine multiple candidate words according to the image encoding information and the multi-head attention mechanism model; When the number of the candidate words is greater than a preset sampling value, taking the plurality of candidate words as target candidate words; In a case where the candidate probability corresponding to the target candidate word is greater than a preset sampling probability, the image description information is determined according to the target candidate word.

4. The multimodal question answering method according to claim 2, characterized in that: The step of inputting the image information into a pre-trained optical character recognition model to obtain the image text information includes: Performing character recognition processing on the image information according to the optical character recognition model to obtain a plurality of initial recognition characters; The plurality of initially recognized characters are matched with a preset recognition database to obtain the image text information.

5. The multimodal question answering method according to claim 1, characterized in that: The converting process of the table information to obtain the table text information includes: Determine the number of rows and columns, table header name and table position information from the table information; The table information is formatted according to the number of rows and columns, the table header name and the table position information to obtain the table text information.

6. The multimodal question answering method according to claim 1, characterized in that: After inputting the question-related information into the pre-trained self-attention mechanism model to obtain the question answering result, the method further includes: Determine a question answer result identifier according to the user question; Determine the answer loss value according to the question answering result and the question answering result identifier; The self-attention mechanism model is subjected to parameter adjustment processing according to the answer loss value.

7. The multimodal question answering method according to claim 1, characterized in that: The determining of context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text includes: Determine image feature information according to the image text information and the image description information; The image feature information, the table text information, the initial text information and the relationship category text are combined and processed to obtain the context information.

8. A multimodal question-answering device, characterized in that: include: A first processing module, used for acquiring image information, table information and initial text information; A second processing module is used to extract features from the image information to obtain image text information and image description information, and to convert the table information to obtain table text information; A third processing module is used to input the image information into a pre-trained image feature extraction model to obtain an image feature vector, and to input the table text information and the initial text information into a pre-trained text feature extraction model to obtain a text feature vector; A fourth processing module, used for inputting the image feature vector and the text feature vector into a pre-trained relationship generation model to obtain a relationship category text; a fifth processing module, configured to determine context information according to the image text information, the image description information, the table text information, the initial text information and the relationship category text; A sixth processing module, configured to combine the context information with the acquired user question to obtain question-related information; The seventh processing module is used to input the question-related information into a pre-trained self-attention mechanism model to obtain a question-answering result.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multimodal question-answering method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the multimodal question answering method described in any one of claims 1 to 7.

Citation Information

Cited By

  • Multimodal fusion question and answer method and device, electronic equipment and storage medium

    CN119938859A

  • Robot question and answer response generation method and device, computer equipment, readable storage medium and program product

    CN120014417A

  • Robot question and answer response generation method, device, computer device, readable storage medium and program product

    CN120014417B

  • Data processing method and device, storage medium and electronic equipment

    CN120910291A