Reading Comprehension Method, Device, Storage Medium and Computer Equipment

Abstract

The present application discloses a reading comprehension method, apparatus, storage medium, and computer device. The method includes: obtaining text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; extracting a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options; extracting a picture vector representation of the image data; calculating a multi-modal vector representation containing text information and image information according to the text vector representation and the picture vector representation; calculating a probability value of each option as the correct answer according to the multi-modal vector representation, so as to determine the correct answer matching the question and the scene picture from the options, which can achieve multi-modal reading comprehension of picture and text question input and improve the correctness of reading comprehension.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a reading comprehension method, apparatus, storage medium, and computer device. Background Art

[0002] Currently, there are few reading comprehension methods and systems for multi-modal. Basically, they are all single-modal models that process data, extract relevant features, and apply them to solve problems that require multiple modalities to solve simultaneously. For multi-modal reading comprehension, the common current practice is to extract useful information in the picture through image recognition (such as OCR, face recognition, human action recognition, etc.) as features, and then use these features through models for processing text (such as GRU, LSTM, CNN, BERT, etc. models) to model the problem, and then combine the image features and text features to obtain the answer to the problem by calculating the similarity of the feature vectors. To process multi-modal reading comprehension problems, existing methods all require multiple systems to cooperate to completely solve the multi-modal reading comprehension problem. It is necessary to simultaneously use an image recognition system and a natural language processing system to synchronously process the two signals, and then build a third system based on the results of the two systems to uniformly complete the calculation of the final answer. Currently, relevant multi-modal reading comprehension systems are relatively complex when processing multi-modal signal inputs and finally completing reading comprehension problems, and are prone to losing mutually increasing signals, resulting in a high error rate and low correctness of reading comprehension. Summary of the Invention

[0003] Embodiments of this application provide a reading comprehension method, apparatus, storage medium, and computer device, which can achieve multi-modal reading comprehension of picture and text question inputs, and improve the correctness of reading comprehension.

[0004] On the one hand, a reading comprehension method is provided, and the method includes:

[0005] Obtain text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture;

[0006] Extract a text vector representation of the text data, where the text vector representation includes text information of the question and text information of the options;

[0007] Extract a picture vector representation of the image data;

[0008] Calculate a multi-modal vector representation including text information and image information according to the text vector representation and the picture vector representation;

[0009] Based on the multi-modal vector representation, calculate the probability value of each of the options being the correct answer, so as to determine the correct answer that matches the question and the scenario picture from the options according to the probability value.

[0010] Optionally, the extraction of the text vector representation of the text data includes:

[0011] Convert each word in the text data into the corresponding serial number of the word in the vocabulary through the vocabulary, and find the text vector representation of the text data according to the serial number.

[0012] Optionally, the extraction of the picture vector representation of the image data includes:

[0013] Perform object detection and feature extraction on the scenario picture according to the object detection model to obtain the picture vector representation, where the picture vector representation includes the image information vector representation of each visual object in the scenario picture and the image information vector representation of the entire picture.

[0014] Optionally, the calculation of the multi-modal vector representation including text information and image information according to the text vector representation and the picture vector representation includes:

[0015] Process the text vector representation and the picture vector based on the self-attention model to obtain the global interaction information among the text information of the question, the text information of the options, and the image information;

[0016] Perform normalization processing on the global interaction information to obtain the first normalized information;

[0017] Determine the multi-modal vector representation including text information and image information according to the global interaction information and the first normalized information.

[0018] Optionally, the process of processing the text vector representation and the picture vector based on the self-attention model to obtain the global interaction information among the text information of the question, the text information of the options, and the image information includes:

[0019] Input the embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculate the matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation;

[0020] Determine the global interaction information among the text information of the question, the text information of the options, and the image information according to the product of the matching matrix and the embedding vector representation.

[0021] Optionally, determining a multi-modal vector representation including text information and image information according to the global interaction information and the first normalization information includes:

[0022] Adding the global interaction information and the first normalization information to obtain first summation information;

[0023] Inputting the first summation information into a fully connected layer for processing, and then performing normalization processing on the output result of the fully connected layer to obtain second normalization information;

[0024] Adding the first summation information and the second normalization information to obtain the multi-modal vector representation including text information and image information.

[0025] Optionally, the method further includes:

[0026] Obtaining a position vector representation and a type vector representation, where the position vector representation is used to label the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type;

[0027] The calculating a multi-modal vector representation including text information and image information according to the text vector representation and the picture vector representation includes:

[0028] Calculating a multi-modal vector representation including text information and image information according to the text vector representation, the picture vector representation, the position vector representation, and the type vector representation.

[0029] Optionally, the calculating a probability value of each option as the correct answer according to the multi-modal vector representation, and determining the correct answer matching the question and the scenario picture from the options according to the probability value includes:

[0030] Performing a splitting process on the multi-modal vector representation to obtain a question option representation and an image option representation;

[0031] Processing the question option representation and the image option representation based on a cross-attention model to obtain an attention point vector representation;

[0032] Calculating a probability value of each option as the correct answer according to the attention point vector representation, and determining the correct answer matching the question and the scenario picture from the options according to the probability value.

[0033] On the other hand, a reading comprehension device is provided, and the device includes:

[0034] An acquisition unit for acquiring text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture;

[0035] A first extraction unit for extracting a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options;

[0036] A second extraction unit for extracting a picture vector representation of the image data;

[0037] A calculation unit for calculating a multimodal vector representation containing text information and image information based on the text vector representation and the picture vector representation;

[0038] A determination unit for calculating a probability value of each option as the correct answer based on the multimodal vector representation, so as to determine the correct answer matching the question and the scene picture from the options according to the probability value.

[0039] On the other hand, a computer-readable storage medium is provided, where the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the reading comprehension method described in any of the above embodiments.

[0040] On the other hand, a computer device is provided, where the computer device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the steps in the reading comprehension method described in any of the above embodiments by calling the computer program stored in the memory.

[0041] On the other hand, a computer program product is provided, including computer instructions, and when the computer instructions are executed by a processor, the steps in the reading comprehension method described in any of the above embodiments are implemented.

[0042] In the embodiments of the present application, by obtaining the text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; extracting the text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options; extracting the picture vector representation of the image data; calculating the multi-modal vector representation containing text information and image information according to the text vector representation and the picture vector representation; calculating the probability value of each option as the correct answer according to the multi-modal vector representation, so as to determine the correct answer that matches the question and the scene picture from the options. In the embodiments of the present application, the multi-modal reading comprehension of picture and text questions is realized through the Transformer model. The model calculates the multi-modal vector representation containing text information and image information by simultaneously inputting the image data and the text data including the question and options through the attention mechanism inside the Transformer model, so as to filter out the useful picture information and text information, and then select the correct answer option according to the multi-modal vector representation, improving the correctness of reading comprehension. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0044] Figure 1 It is the structural framework diagram of the reading comprehension system provided by the embodiments of the present application.

[0045] Figure 2 It is the first flow schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0046] Figure 3 It is the first application scenario schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0047] Figure 4 It is the second flow schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0048] Figure 5 It is the second application scenario schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0049] Figure 6 It is the third flow schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0050] Figure 7 It is the third application scenario schematic diagram of the reading comprehension method provided by the embodiments of the present application.

[0051] Figure 8 It is a schematic structural diagram of the reading comprehension device provided by the embodiment of the present application.

[0052] Figure 9 It is a schematic structural diagram of the computer device provided by the embodiment of the present application. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0054] The embodiment of the present application provides a reading comprehension method, device, computer device and storage medium. Specifically, the reading comprehension method of the embodiment of the present application can be executed by a computer device, where the computer device can be a terminal or a server, etc. The terminal can be a smart phone, a tablet computer, a notebook computer, a smart TV, a smart speaker, a wearable smart device, a personal computer (Personal Computer, PC), etc. The terminal can also include a client, and the client can be a video client, a browser client or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0055] The embodiment of the present application can be applied to various scenarios such as artificial intelligence, speech recognition, and intelligent transportation.

[0056] First, some nouns or terms that appear in the process of describing the embodiment of the present application are explained as follows:

[0057] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling machines to have the functions of perception, reasoning, and decision-making. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0058] Machine Learning (ML) is an interdisciplinary field that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning.

[0059] Deep Learning (DL) is a branch of machine learning and is an algorithm that attempts to use multiple processing layers containing complex structures or composed of multiple non-linear transformations to perform high-level abstractions on data. Deep learning is to learn the internal laws and representation levels of training sample data, and the information obtained during these learning processes is very helpful for the interpretation of data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have the ability to analyze and learn like humans and be able to recognize data such as text, images, and sounds. Deep learning is a complex machine learning algorithm, and the effects achieved in speech and image recognition far exceed previous related technologies.

[0060] A Neural Network (NN) is a deep learning model in the fields of machine learning and cognitive science that mimics the structure and function of biological neural networks.

[0061] The Transformer model is a classic NLP (Natural Language Processing) model. The Transformer model encodes the input and calculates the output entirely based on attention, without relying on recurrent neural networks or convolutional neural networks for sequence alignment. The Transformer model uses the Self-Attention mechanism instead of the sequential structure of RNNs, enabling parallel training of the model and allowing it to have global information.

[0062] Currently, when relevant multi-modal reading comprehension systems process multi-modal signal inputs and finally complete reading comprehension questions, there are mainly the following deficiencies:

[0063] 1. For multi-modal tasks, the method of processing by multiple systems can lead to error accumulation. For example, if the image system makes an error and extracts useless feature signals, it will further affect the subsequent systems to make correct judgments. Similarly, when the system for processing text makes an error or its accuracy decreases, then when answering reading comprehension questions based on picture information later, it will be more likely to make incorrect judgments.

[0064] 2. When multi-modal signals are jointly processed using multiple single-modal systems, some mutually enhancing signals will be lost. Because when extracting features, the image system does not know what features the text system needs, and the text system cannot obtain the signal input of the image when extracting features, so some important information may not be obtained properly during the final fusion.

[0065] 3. When multiple systems are used to process information of different modalities separately, the system becomes more complex, and thus the probability of system errors will increase significantly. Since physically the image and text are processed separately, some ambiguities that can be eliminated through co-occurrence relationships between the text and the image are lost, resulting in a significant decline in the ability to answer reading comprehension questions about multi-modal signals. And reading comprehension questions themselves are extremely difficult. If the information of the two modalities cannot be modeled simultaneously, the difficulty will be further increased.

[0066] With the development of artificial intelligence technology, single-modal models are increasingly unable to meet the needs of daily human-computer interaction. To comply with the requirements of technological development, the embodiments of this application propose a multi-modal reading comprehension system. When solving reading comprehension problems, the object of reading comprehension in the embodiments of this application changes from the input text to pictures of complex scenes. By understanding the pictures, the correct answers are then selected. With the development of artificial neural networks, the ability of a single model to process multiple modal information has increasingly become a key technology for human-computer interaction. Therefore, the embodiments of this application propose a multi-modal reading comprehension technology to help machines better understand environmental signals and answer relevant questions based on these signals. This technology has great application potential in future robots.

[0067] The embodiments of this application can be used to solve multi-modal reading comprehension problems or image Q&A problems with picture and text questions as input. It can be widely applied to human-machine real-scene conversations, as well as image understanding and retrieval. It has very great application value.

[0068] The multi-modal reading comprehension problem-solving system based on the deep learning model of neural network proposed in the embodiments of this application can just well overcome the above several problems. First, the embodiments of this application perform vectorized representation on text data, and each word, each question, each option, and each sentence will be mapped to a specific space. Then, the representation of the option will perform attention calculation based on spatial relationship with the question and the image vectors of different modalities, that is to say, the question and the option will obtain the most relevant part in the image information through this calculation. Finally, the fully connected deep neural network will map the relevant information between the question image and the option to the solution space, so as to select a correct option as the answer. The main advantages of the system in the embodiments of this application are: 1. It has good migration ability. As long as the training data is replaced, it can be migrated to reading comprehension problems of other modalities. 2. It does not require a large amount of expert knowledge to be manually edited. The model automatically obtains the knowledge for problem-solving from the data. 3. The image questions and options used are represented by spatial vectors, and it has strong generalization ability. 4. One set of system can process inputs of both image and text modalities. It greatly simplifies the later application deployment.

[0069] Please refer to Figure 1 , Figure 1This is the structural framework diagram of the reading comprehension system provided by the embodiments of this application. The reading comprehension system includes a multi-modal Transformer model. First, text data in the text modality is input, which includes questions and options. Then, image data in the image modality is input, and the image data includes scene pictures. Then there is a multi-modal Transformer model. The option that matches the question and the scene picture can be calculated through the Transformer model as the correct answer to the question. Among them, the Transformer model filters out useful picture information and text information through the attention mechanism inside the Transformer model by simultaneously inputting the signals of the question, options, and scene pictures, calculates the multi-modal vector representation containing text information and image information to filter out useful picture information and text information, and then selects the correct answer option through a fully connected layer based on this filtered information. Among them, the feature representations of the two modalities of text and image are learned when the picture and text are input simultaneously. The model can accurately obtain useful multi-modal vector representation information and finally accurately determine whether this scene picture matches the answer to the current question. Since the entire reading comprehension system directly completes the reading comprehension problem through a single model, the structure is relatively simple and the performance is relatively efficient.

[0070] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0071] Each embodiment of this application provides a reading comprehension method, which can be executed by a terminal or a server, or jointly executed by a terminal and a server; this application takes the reading comprehension method being executed by the server as an example for illustration.

[0072] Please refer to Figures 2 to 7 , Figure 2 , Figure 4 and Figure 6 are all the flow schematic diagrams of the reading comprehension method provided by the embodiments of this application, Figure 3 , Figure 5 and Figure 7 are all the application scenario schematic diagrams of the reading comprehension method provided by the embodiments of this application. The method includes:

[0073] Step 110, obtain the text data and image data to be processed, where the text data includes a question and the options corresponding to the question, and the image data includes a scene picture.

[0074] For example, the text data includes a question and options corresponding to the question, where the options can be one or more. For instance, the question to be processed obtained is "Who wears glasses in the picture?"; if there is one option, for example, the option is "man", and if there are multiple options, for example, the options are "man, woman, child", etc.

[0075] For example, the image data includes a scene picture, which is a picture provided for the scene corresponding to the problem description. For example, during the answering process, the user can observe this scene picture to answer the question.

[0076] Step 120: Extract the text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options.

[0077] Optionally, the extraction of the text vector representation of the text data includes:

[0078] Convert each word in the text data into the corresponding serial number of each word in the vocabulary through the vocabulary, and find the text vector representation of the text data according to the serial number.

[0079] First, the text data can be vectorized to map each word, each question, each option, and each sentence to a specific space to obtain the text vector representation of the text data.

[0080] For example, in combination with Figure 3 it is described as follows. The reading comprehension system can include a multi-modal data input processing module, a multi-modal feature extraction module, and an option scoring module. Among them, the feature extraction module can adopt a Transformer model. The TXT format text data (i.e., the original text corresponding to the question and options in TXT format) input into the data input processing module is converted into the serial number (ID) of each word corresponding to the vocabulary through the vocabulary, and then the original text is used to find the embedding vector representation of each word in the vocabulary through the ID. For example, for the question "Who wears glasses in the picture?" and the option "man", it is converted into ID [1, 4, 3, 6, 7, 0, 12, 87, 98, 10] through the vocabulary. Since the ID of "picture" in the vocabulary is 1, it is converted into 1, and then the vector representation (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10) corresponding to each word is found through the ID to obtain a sequence of word vectors. Then, this sequence of word vectors is used as the input parameter of the Transformer model later, and can be defined as the text vector representation. The text vector representation is a matrix of seq_len multiplied by hid_size, where seq_len represents the text length and hid_size represents the size of the word vector.

[0081] Step 130: Extract the picture vector representation of the image data.

[0082] Optionally, the extraction of the picture vector representation of the image data includes:

[0083] Perform object detection and feature extraction on the scene picture according to the object detection model to obtain the picture vector representation, where the picture vector representation includes the image information vector representation of each visual object in the scene picture and the image information vector representation of the entire picture.

[0084] For example, for the image data input to the data processing module, first extract the physical numerical information through the object detection model (FastRCNN), and then through model learning, obtain a FastRCNN model that can correctly extract physical information. Fast RCNN (Fast Regions with CNN features) is a fast region-based convolutional network method for object detection.

[0085] For example, in combination with Figure 3 For illustration, for the image data containing the scene picture, first perform object detection and feature extraction on the scene picture in the image data through Fast-RCNN to obtain the image information vector representation of each visual object in the picture and the image information vector representation of the entire picture. Among them, the visual object corresponds to the object that needs to be concerned in the text data. For example, if the text data mentions "who in the picture", then it is necessary to pay attention to the people in the scene picture, such as the men and women in the picture. Among them, the image information vector representation of the entire picture will be applied to each text character respectively, and the image information vector representation of a single visual object will correspond to the text vector specially representing the image information, that is Figure 3 the img text vector shown. The final image data is also a matrix of seq_len multiplied by hid_size as the input parameter of the Transformer model, which can be defined as the picture vector representation.

[0086] Step 140: Calculate the multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation.

[0087] For example, input the Embedding vector representations containing images, questions, and options into the Transformer model to calculate the multimodal vector representations between each option and the images and questions. The multimodal vector representation is a vector that contains both text information and image information.

[0088] Optionally, the method further includes: obtaining a position vector representation and a type vector representation, where the position vector representation is used to label the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type;

[0089] Calculating a multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation includes:

[0090] Calculating a multimodal vector representation containing text information and image information according to the text vector representation, the picture vector representation, the position vector representation, and the type vector representation.

[0091] Among them, the position vector representation is used to label the position of each word in the text data, and the size of the position vector representation is a matrix of seq_len multiplied by hid_size. The type vector representation is used to distinguish between text type and image type, and the size of the type vector representation is a matrix of seq_len multiplied by hid_size. For example, the text type is represented as 0, and the image type is represented as 1.

[0092] For example, as Figure 3 shown, the input parameters finally input into the Transformer model may include an embedding vector representation Embedding composed of the text vector representation plus the picture vector representation plus the position vector representation and the type vector representation, denoted as E. For example, input the Embedding vector representation containing images, questions, and options into the Transformer model to calculate the multimodal vector representation between each option and the pictures and questions. The multimodal vector representation is a vector that contains both text information and image information, that is, the vector representation of each in the option and the question and the representation of the image are calculated through the internal calculation of the transformer model, and the optimal feature vectors are extracted through the co-occurrence of each group. Finally, connect this multimodal vector representation to a fully connected layer to calculate the correct answer to the current question. Among them, the fully connected layer finally outputs the probability value of each option becoming the correct answer, and finally normalize these probability values as the probability of each option being the correct answer.

[0093] For example, in combination with Figure 3 it can be explained that the text vector representation and the picture vector representation can be processed by a multimodal feature extraction module to calculate a multimodal vector representation containing text information and image information. Among them, the feature extraction module can adopt a Transformer model. The main function of the feature extraction module is to calculate a multimodal vector representation that fuses text information and image information. As Figure 3As shown, after being processed by the Transformer model, a multi-modal vector representation that combines text information and image information will ultimately be obtained, and this multi-modal vector representation can be used to calculate the probability value of whether the question answer matches the current scene picture. As Figure 3 shown, what is calculated is the probability value of the option "man" being the correct answer. This feature extraction module is mainly used to calculate the correlation relationship among the image, the question, and the option, and output a matching feature matrix, which can be used as a multi-modal vector representation containing text information and image information.

[0094] Optionally, as Figure 4 shown, step 140 can be implemented through steps 141 to 143, specifically:

[0095] Step 141, process the text vector representation and the picture vector based on the self-attention model to obtain the global interaction information among the text information of the question, the text information of the option, and the image information.

[0096] Optionally, the process of processing the text vector representation and the picture vector based on the self-attention model to obtain the global interaction information among the text information of the question, the text information of the option, and the image information includes: input the embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculate the matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; determine the global interaction information among the text information of the question, the text information of the option, and the image information according to the product of the matching matrix and the embedding vector representation.

[0097] Among them, the length dimension of the global interaction information is the same as the length dimension of the embedding vector representation.

[0098] For example, please refer to Figure 5 , the input of the feature extraction module is: the vector representations of the image, the question, and the option, which is a matrix with a size of seq_len multiplied by hid_size. For example, the vector representations of the image, the question, and the option can be an Embedding vector representation composed of the text vector representation plus the picture vector representation, denoted as E. The vector representations of the image, the question, and the option can also be an Embedding vector representation composed of the text vector representation plus the picture vector representation plus the position vector representation and the type vector representation, denoted as E.

[0099] The output of the feature extraction module is: a multi-modal vector representation that combines image information and all text information, which is a matrix with a size of seq_len multiplied by hid_size.

[0100] For example, please refer toFigure 5 , calculate the matching matrix through the self-attention model inside the feature extraction module. The input is the embedding vector representation E of seq_len multiplied by hid_size, and this embedding vector representation is the vector representation of the image, question, and options; the output is the global interaction information H s , H s The size of H is seq_len multiplied by hid_size. In the specific calculation process, self_attention calculates the attention representation with itself, and the E matrix multiplies E T , to obtain the matching matrix M. Among them, the size of the matching matrix M is seq_len multiplied by seq_len, and then the M matrix multiplies E to obtain H s , H s The size of H is seq_len multiplied by hid_size. Among them, E T is the transpose matrix of the E matrix.

[0101] Step 142, perform normalization processing on the global interaction information to obtain the first normalized information.

[0102] For example, please refer to Figure 5 , perform normalization on the global interaction information H output by the self_attention model s , that is, norm, to obtain the first normalized information H n , the first normalized information H n The size of H is seq_len multiplied by hid_size, and normalization does not affect the matrix size. Among them, the length dimension of the global interaction information is the same as the length dimension of the embedding vector representation, and the word vector size of the global interaction information is the same as the word vector size of the embedding vector representation.

[0103] Step 143, determine the multi-modal vector representation containing text information and image information according to the global interaction information and the first normalized information.

[0104] Optionally, the determining the multi-modal vector representation containing text information and image information according to the global interaction information and the first normalized information includes: adding the global interaction information and the first normalized information to obtain the first summation information; inputting the first summation information into a fully connected layer for processing, and then performing normalization processing on the output result of the fully connected layer to obtain the second normalized information; adding the first summation information and the second normalized information to obtain the multi-modal vector representation containing text information and image information.

[0105] For example, please refer to Figure 5 , the input of the fully connected layer is Hs +H n , that is, adding the global interaction information H s to the first normalized information H n to obtain the first summation information, and then inputting the first summation information into a fully connected layer for processing, where the output result of the fully connected layer has a size of seq_len multiplied by hid_size. Then, normalize the output result of the fully connected layer (norm) to obtain the second normalized information, and add the obtained second normalized information to the first summation information again to obtain the multi-modal vector representation H nn . Since H nn has the same size as the input E matrix, multiple layers of the self-attention model can be stacked here, usually set to 12 layers or 24 layers.

[0106] For example, the multi-modal vector representation H nn Finally, directly output output, the size of output is a matrix of seq_len multiplied by hid_size, and this multi-modal vector representation is a multi-modal vector representation that fuses image information and all text information. Among them, the length dimension of the multi-modal vector representation is the same as that of the embedding vector representation.

[0107] Through this feature extraction module, a multi-modal vector representation that fuses options, images, and questions is calculated. This multi-modal vector representation contains a highly abstract semantic matching relationship between text and images, providing rich information for the subsequent module to select the correct answer based on the matching information. At the same time, it also transforms the previous simple character-level text matching into a matching between vector spaces. This feature extraction module elevates text matching to the semantic space level.

[0108] Step 150, according to the multi-modal vector representation, calculate the probability value of each option as the correct answer, so as to determine the correct answer that matches the question and the scenario picture from the options according to the probability value.

[0109] For example, the output result (the multi-modal vector representation that fuses options, images, and questions) of the output of the feature extraction module can be processed by an option scoring module to calculate the probability value of the current option as the correct answer.

[0110] Optionally, as Figure 6 shown, step 150 can be implemented through steps 151 to 153, specifically:

[0111] Step 151, split the multi-modal vector representation to obtain a question option representation and an image option representation.

[0112] For example, please refer to Figure 7 The input multi-modal vector representation that combines images and text is split by a matrix to obtain the question option representation H qc and the image option representation H pc . The size of Output is a matrix of seq_len times hid_size, the question option is a matrix of seq1_len times hid_size, and the image option representation is a matrix of sseq2_len times hid_size, where seq = seq1 + seq2.

[0113] Step 152, process the question option representation and the image option representation based on the cross-attention model to obtain the focus point vector representation.

[0114] For example, please refer to Figure 7 , based on the cross-attention model, use the question option representation H qc to calculate an attention vector representation from the image option representation H pc . H qc and H pc are respectively connected with a fully connected layer to obtain H’ qc and H’ pc , then multiply the H’ qc matrix by the transpose of H’ pc to obtain the attention representation, and the matrix size of the attention representation is seq1 times seq2. The attention representation is then multiplied by the matrix of H pc [[ID=3,2]]to obtain the attention vector representation H att , and the matrix size of H att is seq1 times hid_size. This attention representation is the content or point of interest, such as Figure 3 the example question and the scene picture, and the obtained focus point vector representation can include the content of "the man in the picture wears glasses" and "the woman in the picture does not wear glasses".

[0115] Step 153, calculate the probability value of each option as the correct answer according to the focus point vector representation, so as to determine the correct answer that matches the question and the scene picture from the options.

[0116] For example, please refer to Figure 7, the final calculation option is the probability value of the answer. The Attend vector represents the sum in the seq1 dimension to obtain vector V, and the size of vector V is a 1 by hid_size vector. Then, a fully connected layer is connected to transform vector V into a score, which can represent the probability value of the option being the correct answer.

[0117] Among them, when calculating the score through the option scoring module, the options, questions, and pictures are treated differently again, and through the attention method again, the probability value of the current option being the correct answer is finally obtained. If a question has four options, the correct answer can be determined by comparing the different scores of the four options. For example, the option with the highest probability value (i.e., the highest score) is determined as the correct answer that matches the question and the scenario picture.

[0118] When solving problems, the entire multi-modal reading comprehension system processes through the multi-modal data input processing module, the multi-modal feature extraction module, and the option scoring module in sequence to obtain the correct answer to the question. Among them, before using this reading comprehension system, sufficient multi-modal reading comprehension data can also be provided to perform model learning on this reading comprehension system. After learning, the entire reading comprehension system can automatically answer reading comprehension questions in a specific scenario (such as the English part of the high school entrance examination and college entrance examination).

[0119] The embodiments of this application can be based on the multi-modal reading comprehension technology of the transformer, have the ability to extract various modal information in multiple modalities, and can calculate the correct answer to the question according to the multi-modal vector representation.

[0120] The embodiments of this application can adopt the end-to-end deep neural network technology. Compared with the traditional rule- and expert knowledge-based system, this reading comprehension system greatly simplifies the entire problem-solving process and eliminates the complicated rules and the process of knowledge feature extraction. Only by providing the data for system model training can this system automatically learn the knowledge and rules related to problem-solving. This reading comprehension system can be quickly migrated to other similar tasks and fields. This reading comprehension system adopts a modular design scheme, and different modules have specific functions and can be quickly migrated to related systems. For example, the option scoring module can be directly applied to other tasks that require calculating vector similarity. And the entire system can be directly applied to multi-modal reading comprehension tasks by replacing the training data.

[0121] Any combination of the above all technical solutions can form the optional embodiments of this application, which will not be elaborated one by one here.

[0122] In an embodiment of the present application, by obtaining text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; extracting a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options; extracting a picture vector representation of the image data; calculating a multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation; calculating a probability value for each option as the correct answer according to the multimodal vector representation, so as to determine the correct answer matching the question and the scene picture from the options. In an embodiment of the present application, a multimodal reading comprehension is implemented through a Transformer model. The model calculates a multimodal vector representation containing text information and image information by simultaneously inputting image data and text data including a question and options, through an attention mechanism inside the Transformer model, so as to filter out useful picture information and text information, and then selects the correct answer option according to the multimodal vector representation, improving the correctness of reading comprehension.

[0123] To facilitate better implementation of the reading comprehension method in the embodiment of the present application, the embodiment of the present application further provides a reading comprehension device. Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of the reading comprehension device provided in the embodiment of the present application. Among them, the reading comprehension device 200 may include:

[0124] An acquisition unit 201, configured to acquire text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture;

[0125] A first extraction unit 202, configured to extract a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options;

[0126] A second extraction unit 203, configured to extract a picture vector representation of the image data;

[0127] A calculation unit 204, configured to calculate a multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation;

[0128] A determination unit 205, configured to calculate a probability value for each option as the correct answer according to the multimodal vector representation, so as to determine the correct answer matching the question and the scene picture from the options.

[0129] Optionally, the first extraction unit 202 may be configured to convert each word in the text data into a corresponding serial number of the word in the vocabulary through the vocabulary, and search for a text vector representation of the text data according to the serial number.

[0130] Optionally, the second extraction unit 203 may be configured to perform object detection and feature extraction on the scene picture according to a target detection model to obtain the picture vector representation, where the picture vector representation includes an image information vector representation of each visual object in the scene picture and an image information vector representation of the entire picture.

[0131] Optionally, the calculation unit 204 may specifically be configured to: process the text vector representation and the picture vector based on a self-attention model to obtain global interaction information among the text information of the question, the text information of the option, and the image information; perform normalization processing on the global interaction information to obtain first normalized information; and determine a multimodal vector representation including text information and image information according to the global interaction information and the first normalized information.

[0132] Optionally, when the calculation unit 204 processes the text vector representation and the picture vector based on a self-attention model to obtain global interaction information among the text information of the question, the text information of the option, and the image information, it may be configured to: input an embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculate a matching matrix according to a product between the embedding vector representation and a transposed matrix of the embedding vector representation; and determine the global interaction information among the text information of the question, the text information of the option, and the image information according to a product between the matching matrix and the embedding vector representation.

[0133] Optionally, when the calculation unit 204 determines a multimodal vector representation including text information and image information according to the global interaction information and the first normalized information, it may be configured to: add the global interaction information and the first normalized information to obtain first summation information; input the first summation information into a fully connected layer for processing, and then perform normalization processing on an output result of the fully connected layer to obtain second normalized information; and add the first summation information and the second normalized information to obtain the multimodal vector representation including text information and image information.

[0134] Optionally, the acquisition unit 201 may further be configured to acquire a position vector representation and a type vector representation, where the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between a text type and an image type;

[0135] The computing unit 204 can be used to calculate a multi-modal vector representation containing text information and image information according to the text vector representation, the picture vector representation, the position vector representation, and the type vector representation.

[0136] Optionally, the determining unit 205 can be specifically configured to: split the multi-modal vector representation to obtain a question option representation and an image option representation; process the question option representation and the image option representation based on a cross-attention model to obtain a focus point vector representation; calculate a probability value of each option being the correct answer according to the focus point vector representation, so as to determine the correct answer that matches the question and the scenario picture from the options according to the probability value.

[0137] It should be noted that the functions of the various modules in the reading comprehension device 200 in the embodiments of the present application can be correspondingly referred to the specific implementation manners of any of the above method embodiments, and will not be elaborated here.

[0138] Each of the above units in the reading comprehension device can be implemented in whole or in part by software, hardware, and their combination. Each of the above units can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so that the processor can call and execute the operations corresponding to each of the above units.

[0139] The reading comprehension device 200 can be integrated, for example, in a terminal or a server with a storage and a processor and having computing capabilities, or the reading comprehension device 200 is the terminal or the server. The terminal can be a smart phone, a tablet computer, a notebook computer, a smart TV, a smart speaker, a wearable smart device, a personal computer (PC), etc. The terminal can also include a client, and the client can be a video client, a browser client, or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0140] Figure 9 It is a schematic structural diagram of a computer device provided by an embodiment of the present application, as Figure 9As shown in the figure, the computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, the memory 302, and the processor 303 communicate with each other through the communication bus 304. The communication interface 301 is used for the device 300 to communicate with external devices. The memory 302 can be used to store software programs and modules. The processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0141] Optionally, the processor 303 may call the software programs and modules stored in the memory 302 to perform the following operations: obtaining text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; extracting a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options; extracting a picture vector representation of the image data; calculating a multi-modal vector representation including text information and image information according to the text vector representation and the picture vector representation; calculating a probability value of each option as the correct answer according to the multi-modal vector representation, so as to determine the correct answer that matches the question and the scene picture from the options according to the probability value.

[0142] Optionally, the computer device 300 is the terminal or the server. The terminal may be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a personal computer, or the like. The server may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0143] Optionally, the present application also provides a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the foregoing method embodiments are implemented.

[0144] The present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a computer device, and the computer program enables the computer device to execute the corresponding processes in the reading comprehension method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0145] The present application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the corresponding processes in the reading comprehension method in the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0146] The present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the corresponding processes in the reading comprehension method in the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0147] It should be understood that the processor in the embodiments of the present application may be an integrated circuit chip with the ability to process signals. In the implementation process, the steps of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0148] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0149] It should be understood that the above-mentioned memory is by way of example but not limitation. For example, the memory in the embodiments of the present application can also be a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not be limited to these and any other suitable types of memory.

[0150] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0151] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0152] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0155] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or this part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0156] As described above, it is only the specific implementation manner of the present application. However, the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A reading comprehension method, characterized in that, The method includes: Obtaining text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; Extracting a text vector representation of the text data, where the text vector representation includes the text information of the question and the text information of the options; Extracting a picture vector representation of the image data; Obtaining a position vector representation and a type vector representation, where the position vector representation is used to label the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type; Calculating a multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation, including: calculating a multimodal vector representation containing text information and image information according to the text vector representation, the picture vector representation, the position vector representation, and the type vector representation; Calculating a probability value of each of the options as the correct answer according to the multimodal vector representation, so as to determine the correct answer that matches the question and the scene picture from the options according to the probability value.

2. The reading comprehension method according to claim 1, wherein The extracting the text vector representation of the text data includes: Converting each word in the text data into a corresponding serial number in a vocabulary through the vocabulary, and searching for the text vector representation of the text data according to the serial number.

3. The reading comprehension method according to claim 1, wherein The extracting the picture vector representation of the image data includes: Performing object detection and feature extraction on the scene picture according to an object detection model to obtain the picture vector representation, where the picture vector representation includes an image information vector representation of each visual object in the scene picture and an image information vector representation of the entire picture.

4. The reading comprehension method according to claim 1, characterized in that, The calculating a multimodal vector representation containing text information and image information according to the text vector representation and the picture vector representation includes: Processing the text vector representation and the picture vector based on a self-attention model to obtain global interaction information among the text information of the question, the text information of the options, and the image information; Performing normalization processing on the global interaction information to obtain first normalized information; Determining a multimodal vector representation containing text information and image information according to the global interaction information and the first normalized information.

5. The reading comprehension method according to claim 4, characterized in that The processing the text vector representation and the picture vector based on a self-attention model to obtain global interaction information among the text information of the question, the text information of the options, and the image information includes: Inputting an embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculating a matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; Determining the global interaction information among the text information of the question, the text information of the options, and the image information according to the product of the matching matrix and the embedding vector representation.

6. The reading comprehension method according to claim 4, wherein The determining a multimodal vector representation containing text information and image information according to the global interaction information and the first normalized information includes: Add the global interaction information to the first normalized information to obtain first summation information; After inputting the first summation information into a fully connected layer for processing, perform normalization processing on the output result of the fully connected layer to obtain second normalized information; Add the first summation information to the second normalized information to obtain the multi-modal vector representation containing text information and image information.

7. The reading comprehension method according to any one of claims 1-6, characterized in that, The calculating, according to the multi-modal vector representation, a probability value for each of the options to be the correct answer, and determining, according to the probability value, the correct answer that matches the question and the scene picture from the options, includes: Perform a splitting process on the multi-modal vector representation to obtain a question option representation and an image option representation; Process the question option representation and the image option representation based on a cross-attention model to obtain an attention point vector representation; According to the attention point vector representation, calculate a probability value for each of the options to be the correct answer, and determine, according to the probability value, the correct answer that matches the question and the scene picture from the options.

8. A reading comprehension device, characterized in that, The device includes: An acquisition unit, configured to acquire text data and image data to be processed, where the text data includes a question and options corresponding to the question, and the image data includes a scene picture; A first extraction unit, configured to extract a text vector representation of the text data, where the text vector representation includes text information of the question and text information of the options; A second extraction unit, configured to extract a picture vector representation of the image data; The acquisition unit is further configured to acquire a position vector representation and a type vector representation, where the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type; A calculation unit, configured to calculate a multi-modal vector representation containing text information and image information according to the text vector representation and the picture vector representation, including: calculating a multi-modal vector representation containing text information and image information according to the text vector representation, the picture vector representation, the position vector representation, and the type vector representation; A determination unit, configured to calculate a probability value for each of the options to be the correct answer according to the multi-modal vector representation, and determine, according to the probability value, the correct answer that matches the question and the scene picture from the options.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the reading comprehension method according to any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the steps in the reading comprehension method according to any one of claims 1-7 by calling the computer program stored in the memory.

11. A computer program product, comprising computer instructions, characterized in that, The computer instructions, when executed by a processor, implement the steps in the reading comprehension method according to any one of claims 1-7.

Images