Text visual question answering method and system based on multi-source interaction

Abstract

The application discloses a text visual question answering method and system based on multi-source interaction, relates to the technical field of text visual question answering, and comprises the following steps: collecting data information from a data set and constructing false relation pairs; training image encoders, OCR encoders, question encoders and object encoders respectively by using image categories, OCR problems, question contexts and object categories; performing multi-modal contrast learning by using true relation pairs and false relation pairs; aligning multi-modal encoder semantic relations; integrating multi-modal encoders, using iterative decoding to predict answers; and correcting encoder-decoder parameters based on the predicted answers to obtain an optimal model. The application actively fuses information from a picture branch, a question branch and an OCR branch in a multi-source interaction mode. The information of the question branch can provide accurate guidance and help the OCR branch to exclude possible detection errors or recognition errors.

Description

Technical Field

[0001] This invention relates to the field of text visual question answering technology, specifically to a text visual question answering method and system based on multi-source interaction. Background Technology

[0002] We are currently living in an era of rapid information growth, with information not only abundant in quantity but also diverse in type. Common types of information include text, images, audio, and video. Utilizing multiple types of information can improve efficiency and accuracy in solving practical problems. Visual question answering research emerged in this context. While pure text-based question answering models have been studied for many years, their applications are limited and their scope is narrow. Incorporating more intuitive visual information allows for question answering research in real-world scenarios, making it more practical and promising for future applications, such as question-answering systems for the blind, autonomous driving systems, and search methods combining images and questions. This invention focuses on improving the interaction and fusion capabilities of multimodal features under existing multimodal feature engineering conditions. Multimodal feature fusion and interaction capabilities can not only solve visual question answering tasks but also be applied to other multimodal tasks, demonstrating broad applicability. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by this invention is: how to improve the accuracy and efficiency of text-based visual question answering systems. By integrating multiple encoders and decoders, and utilizing various information such as images, OCR text, question context, and object categories, multimodal contrastive learning is performed to predict the answer. The model parameters are then adjusted based on the predicted answer to obtain the optimal model.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a text-based visual question answering method based on multi-source interaction, comprising the following steps:

[0006] Collect data information from the dataset and construct pseudo-relation pairs;

[0007] The image encoder, OCR encoder, question encoder, and object encoder are trained using image category, OCR question, question context, and object category, respectively.

[0008] By performing multimodal contrastive learning on true relation pairs and false relation pairs, the semantic relations of the multimodal encoder are aligned.

[0009] Integrate a multimodal encoder and use iterative decoding to predict the answer;

[0010] Based on the predicted answer, the encoder / decoder parameters are adjusted to obtain the optimal model;

[0011] Training a locally optimal image encoder involves using annotation tools to extract OCR text and object category labels from an image, targeting the first... Extract from the images. indivual text , in ;extract Image categories ,in ;

[0012] Training the encoder involves cropping each image from 256×256 into 32 8×8 partial images, expressed as follows:

[0013]

[0014] in, For the entirety of a certain image, The first part of an image. The second part of an image. This is the thirty-second part of a certain image;

[0015] Each section of images Each image has 3 channels, resulting in 3 × 8 × 8 = 192 pixel values. Each image is unfolded into a one-dimensional array, forming 1 × 192 input data. The entire image has 32 × 192 input data. A 1 × 192 vector with all zero values ​​is defined and concatenated with the unfolded 32 × 192 image data to form the final image input data. The expression is:

[0016]

[0017] in, Information about the unfolded image This is the first part of the information in the unfolded overall image. This is the thirty-second section of information in the unfolded overall image. A 1×192 vector (0, 0, ..., 0) of all zeros, with a length of 192, represents the overall features of the image;

[0018] Input information The total dimension is 33×192. After passing through a linear layer and an activation function layer, the data dimension is expanded to 768 dimensions, as expressed in the following expression:

[0019]

[0020] in, Activation function For linear computation operations;

[0021] Based on the image segmentation process from left to right and top to bottom, each image segment is assigned a sequence number from 0 to 31. An embedding layer converts each image sequence number into a 768-dimensional sequence feature. This sequence feature is then added to the image feature and fed into three consecutive Transformer layers. Each Transformer layer consists of a multi-level self-attention layer and a feedforward neural network layer. The final image layer extracts the overall image features. These features are then processed by a classification layer to predict the image's category. The cross-entropy loss function is used to output the image encoder's prediction, expressed as:

[0022]

[0023] in, It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category;

[0024] Training a locally optimal OCR encoder involves using a sliding window to traverse the entire image from left to right and from top to bottom. The initial size of the sliding window is set to three different sizes, each with a different length and width. Each window is used to traverse the entire image, and the pixels within each window are processed through a convolutional layer and a pooling layer to obtain image features, expressed as:

[0025]

[0026] in, Refers to the image feature value corresponding to the i-th sliding window. For pooling operations, For convolution operations, The pixel value of the image in the i-th sliding window;

[0027] The features of all windows are input into the LSTM encoder and decoder network, and the OCR text of each window is obtained by decoding. The expression is:

[0028]

[0029] in, For Long Short-Term Memory (LSTM) module operation, The image feature value corresponding to the first sliding window. The image feature value corresponding to the (N+1)th sliding window;

[0030] The CTC algorithm is used to correct duplicate characters in OCR text obtained from the LSTM algorithm. The expression is:

[0031]

[0032] in, The characters obtained from decoding, For connection-time classification algorithms, This is the text after removing duplicate characters;

[0033]

[0034] in, It is the sample number. It is the first word One letter;

[0035] Set the length of the OCR text, discard OCR text content that exceeds the preset length, and use empty words to fill in OCR text content that is shorter than the preset length;

[0036] Training a local optimal object encoder involves setting up three sliding windows of different sizes, but each sliding window has the same length and width, and then performing ResNet operations.

[0037] The ResNet operation involves performing a series of convolutions (first, second, third, fourth, and fifth), pooling, fully connected layers, and a second fully connected layer) sequentially on each image within a sliding window to obtain object features. The expression is as follows:

[0038]

[0039] in, For residual network algorithms, The image features of the i-th sliding window;

[0040] The cross-entropy loss function is used to calculate the cross-entropy loss between the predicted and actual values. The expression is as follows:

[0041]

[0042] in, It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category;

[0043] Training a locally optimal question encoder involves using one embedding layer and three encoding layers to obtain question features. The question is expanded to a preset text length, and empty words are used to pad OCR text content shorter than the preset length. Each word in the question is mapped to a sequence number in a vocabulary list. The embedding layer converts each word into a 768-dimensional feature, and three BERT encoding layers are used to extract the question features. The expression is as follows:

[0044]

[0045]

[0046]

[0047] in, It is the input text. It is a mapping operation. It is the mapped text ID. It is an embedding operation. It is an embedded feature. Language model layer It is the third layer of language features;

[0048] Features of each branch are extracted using four pre-trained encoders. 、 、 、 Arbitrarily select two branches for comparative learning to construct true and false relation pairs. The standard value of a true relation pair is 1, and the standard value of a false relation pair is 0. Pair the branches to generate 6 combinations.

[0049] The image branch has I samples, and the question branch has J samples. Calculate the contrast features between the i-th image and the j-th question. Use the cosine angle to represent the degree of separation between the features: a cosine of 0 indicates no correlation between the two features, and a cosine of 1 indicates a positive correlation. The expression is:

[0050]

[0051] in, It is the inverse cosine function. It is the probability score that the i-th image sample and the j-th question sample are related;

[0052] The loss value for contrastive learning is calculated using the binary cross-entropy loss function, and the loss across multiple combinations is balanced. The expression is:

[0053]

[0054]

[0055] The branches are paired to generate 6 combinations. The mean is calculated, and the training calculation is repeated for m iterations to obtain m neural network classification training models and loss function values. The weight vector and bias term of the convolutional neural network model corresponding to the smallest value among the loss function values ​​are selected as the optimal weight vector and optimal bias term to obtain the optimal model.

[0056] Construct prediction samples using samples not used in training, calculate the score for each word using the trained optimal neural network, take the word with the highest score as the answer, calculate the accuracy by comparing the predicted answer with the actual answer, and the prediction is complete.

[0057] As a preferred embodiment of the text visual question answering method based on multi-source interaction described in this invention, wherein: the data is selected from the TextVQA dataset. The first original image. The second... The original color image is denoted as , , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and These represent the width and height of the image, respectively. This indicates the position in the original color image. The pixel values ​​of each pixel are used to generate transformed images through data augmentation methods such as horizontal flipping, translation, random channel sorting, and cropping. All images are cropped to a fixed size, and the transformed images corresponding to all the original images are then used to form a training set. The number of transformed images is [number missing]. The first training set A color image is denoted as ,in , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and Let represent the width and height of the image, respectively. Select M question-answer pairs from the TextVQA dataset, and denote the m-th question as . Let the m-th answer be denoted as Where m and M are positive integers, .

[0058] Another objective of this invention is to provide a text-based visual question answering system based on multi-source interaction. This system integrates various information sources, including image category, OCR question, question context, and object category. It utilizes an image encoder, OCR encoder, question encoder, and object encoder for feature extraction, then aligns the semantic relationships between the multi-modal encoders through multimodal contrastive learning, and finally uses iterative decoding to predict the answer. This method addresses the shortcomings of existing text-based visual question answering systems in handling multi-source information, improving the system's accuracy and efficiency, and enabling it to better understand and process complex text-based visual question answering tasks.

[0059] To address the aforementioned technical problems, this invention provides the following technical solution: a text visual question answering method based on multi-source interaction, comprising: a data processing module, an encoder training module, a multimodal contrastive learning module, a decoding prediction module, and a model optimization and evaluation module.

[0060] The data processing module collects data information from the dataset, performs data cleaning and preprocessing, constructs pseudo-relation pairs, and provides the cleaned and preprocessed data to the encoder training module.

[0061] The encoder training module trains the image encoder, OCR encoder, question encoder, and object encoder respectively using image category, OCR question, question context, and object category, extracts their respective features, and transmits the trained encoders to the multimodal contrast learning module.

[0062] The multimodal contrastive learning module performs multimodal contrastive learning between true relation pairs and false relation pairs to align the semantic relations of the multimodal encoder, and then transmits the aligned semantic relations of the multimodal encoder to the decoding and prediction module.

[0063] The decoding and prediction module integrates a multimodal encoder and uses iterative decoding to predict the answer.

[0064] The model optimization and evaluation module is based on the predicted answer, corrects the encoder and decoder parameters, obtains the optimal model, constructs prediction samples using samples that were not used in training, calculates the accuracy, and completes the prediction.

[0065] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the text visual question answering method based on multi-source interaction as described above.

[0066] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the text visual question answering method based on multi-source interaction as described above.

[0067] The beneficial effects of this invention are as follows: This invention actively fuses information from the image branch, question branch, and OCR branch through multi-source interaction. Information from the question branch provides accurate guidance, helping the OCR branch eliminate potential detection or recognition errors. Simultaneously, the question branch contains information similar to that in the OCR branch, which helps the neural network improve its localization of the target OCR text. This allows it to find the most relevant OCR text among multiple OCR texts, further improving the accuracy of the answer. Information in the image branch and the OCR branch have spatial relationships; by modeling these spatial relationships, the neural network can find a one-to-one correspondence between the information in the image branch and the information in the OCR branch.

[0068] This invention employs a novel OCR recognition method that not only improves the accuracy of OCR text but also enables the recognition of OCR text in different languages. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0070] Figure 1 The overall flowchart of the text visual question answering method based on multi-source interaction provided in the first embodiment of the present invention;

[0071] Figure 2 A schematic diagram of the optimal encoder in the text visual question answering method based on multi-source interaction provided in the first embodiment of the present invention;

[0072] Figure 3 A schematic diagram of an image encoder for a text visual question answering method based on multi-source interaction provided in the first embodiment of the present invention;

[0073] Figure 4 A schematic diagram of an OCR encoder for a text visual question answering method based on multi-source interaction provided in the first embodiment of the present invention;

[0074] Figure 5 A schematic diagram of an object encoder for a text visual question answering method based on multi-source interaction provided in the first embodiment of the present invention;

[0075] Figure 6 The structural diagram of the text visual question answering system based on multi-source interaction provided in the second embodiment of the present invention;

[0076] Figure 7This diagram illustrates the accuracy and speed of the text visual question answering method based on multi-source interaction provided in the third embodiment of the present invention. Detailed Implementation

[0077] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0078] Example 1

[0079] Reference Figures 1-5 This invention provides a text-based visual question answering method based on multi-source interaction, characterized by: collecting data information from a dataset and constructing pseudo-relation pairs; training an image encoder, an OCR encoder, a question encoder, and an object encoder using image category, OCR question, question context, and object category respectively; aligning the semantic relationships of the multimodal encoders through multimodal contrast learning using true and pseudo-relation pairs; integrating the multimodal encoders and using iterative decoding to predict the answer; and correcting the encoder / decoder parameters based on the predicted answer to obtain the optimal model.

[0080] Step ①_1: Select from the TextVQA dataset The first original image. The second... The original color image is denoted as ,in , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and These represent the width and height of the image, respectively. This indicates the position in the original color image. The pixel values ​​of each pixel are then used. Data augmentation methods such as horizontal flipping, translation, random channel sorting, and cropping are then used to generate transformed images, all of which are cropped to a fixed size. A training set is then constructed from the transformed images corresponding to all the original images, with the number of transformed images being [number missing]. The first training set A color image is denoted as ,in , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and These represent the width and height of the image, respectively.

[0081] Step ①_2: Select M question-answer pairs from the TextVQA dataset. Let the m-th question be denoted as... Let the m-th answer be denoted as Where m and M are positive integers, .

[0082] Step ①_3: Use the annotation tool to extract OCR text and object category labels from the image. The annotation tool is a web-based annotation tool developed using HTML and JavaScript. It allows users to drag and drop the mouse to annotate the smallest bounding box of OCR text or objects, and then input text to represent the content of the OCR text or the category of the object. Specifically, for the p-th image, extract... OCR text { },in ;extract Image categories { },in .

[0083] Step ①_4: Train the local optimal image encoder. For example... Figure 2 As shown, each image is cropped from 256×256 into 32 8×8 partial images.

[0084]

[0085] Each section of images Each image has 3 channels, resulting in a total of 3×8×8=192 pixel values. Each image is unfolded into a one-dimensional array, forming 1×192 input data, for a total of 32×192 input data for the entire image. Then, a 1×192 vector with all zero values ​​is defined and concatenated with the unfolded 32×192 image data to form the final image input data.

[0086]

[0087] Input information The total dimensions are 33×192. Then, through a linear layer and an activation function layer, the data dimensions are expanded to 768 dimensions.

[0088] Img' represents the expanded overall image information, with a dimension of 33×192. Img0' represents the expanded image information of Img0, with a dimension of 1×192. The others are also the expanded results of the image information corresponding to their respective indices. Here, "expanded" means: if the RGB value at position (0,0) is (120, 12, 43), and the RGB value at position (0,1) is (23, 136, 17), then the expanded values ​​from (0,0) to (0,1) are (120, 12, 43, 23, 136, 17). This is the result of expanding two pixels. The result of expanding the 64 pixels above is similar; the RGB values ​​are simply added to a single-dimensional array. The Zero value is a 1×192 vector (0, 0, ..., 0) of length 192, consisting of all zeros. The Zero value represents the overall image features; it is currently all zeros, but after subsequent calculations, it will no longer be all zeros and will become meaningful.

[0089]

[0090] Finally, based on the image segmentation process from left to right and top to bottom, each image segment is assigned a sequence number from 0 to 31. An embedding layer is then used to convert the sequence number of each image into a 768-dimensional sequence feature.

[0091] `linear()` performs a linear computation operation, specifically `wx + b`. In the formula, `x` is `Img'`, with dimensions 33 × 192. `w` is the weight matrix, with dimensions 192 × 768, and `b` is the bias matrix, with dimensions 33 × 768. `wx + b` first performs matrix multiplication, then matrix addition, and `sigmoid` is the activation function.

[0092] Then, the indices 0 to 31 are embedded into a 768-dimensional vector. .

[0093]

[0094] Index is the sequence number, and embed() is the embedding operation. This refers to embedded features. The embedding operation is a linear computation that takes integers as input and outputs a 768-dimensional floating-point vector. The feature resulting from adding the index feature and the image feature is input into three consecutive Transformer layers. Each Transformer layer consists of a multi-level self-attention layer and a feedforward neural network layer. The self-attention layer comprises three fully connected layers, a multiplication operation, and a concatenation operation. The fully connected layer represents the linear computation and activation function computation mentioned above. The multiplication operation is matrix multiplication, and the concatenation operation is the joining of matrices. Specifically, it refers to two matrices that are equal in all dimensions except for one; these matrices can then be joined together along that dimension.

[0095] For example, if one matrix has a dimension of 3×4×5 and another matrix has a dimension of 3×2×5, then the total dimension of the two matrices after concatenation is 3×6×5.

[0096]

[0097] in, This refers to the calculations performed by the Transformer layer. Then, the overall image features are extracted from the last image layer.

[0098]

[0099] Here, the value at position 31 represents the original position of the Zero vector; the feature at this position now becomes the overall image feature. The image features are processed by a classification layer to predict the image's category. This classification layer consists of a fully connected layer and a normalization layer. The fully connected layer converts the image features into probability scores representing the image category, while the normalization layer converts these probability scores into values ​​that sum to 1. A cross-entropy loss function is used to make the image encoder's predictions closer to the true values.

[0100]

[0101] It is the value of the loss function. It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category.

[0102] Step ①_5: Train the locally optimal OCR encoder. For example... Figure 3As shown, a sliding window is used to traverse the entire image from left to right and from top to bottom. The initial size of the sliding window is set to three dimensions: 3×6, 6×12, and 12×24. The entire image is traversed using these three different sized sliding windows, and the pixels in each window are processed through a convolutional layer and a pooling layer to obtain image features.

[0103] A convolutional layer refers to a convolution operation, which uses a 3×3 convolutional kernel to extract image features. A pooling layer refers to a pooling operation, which uses a 2×2 pooling kernel to reduce the dimensionality of image features and extract higher-level image features.

[0104]

[0105] in, Refers to the image feature value corresponding to the i-th sliding window. For pooling operations, For convolution operations, The pixel value of the image in the i-th sliding window;

[0106] Subsequently, the features of all windows are input into the LSTM encoder and decoder network to decode and obtain the OCR text of each window.

[0107] The LSTM codec is a general-purpose network model, a variant of recurrent neural networks (RNNs) designed to handle sequential data. It addresses the vanishing and exploding gradient problems inherent in traditional RNNs, making it more suitable for modeling long sequences. Its structure is as follows: Memory Unit: This is the core component of the LSTM network, used to store and maintain information. The memory unit allows the network to remember information from the sequence for extended periods, enabling better processing of long sequences. The memory unit has a hidden state and a cell state, and the information transfer between them is controlled by a gating mechanism. Gating Mechanism: LSTM networks include three gating mechanisms: the forget gate, the input gate, and the output gate. Forget Gate: Determines which information is discarded from the cell state. Input Gate: Determines which new information is added to the cell state. Output Gate: Determines which information is extracted from the cell state and sent to the network's hidden state. Activation Function: The gating mechanism in the LSTM network uses activation functions to control the flow of information, allowing it to learn and adjust the retention and forgetting of information.

[0108]

[0109] in, For Long Short-Term Memory (LSTM) module operation, The image feature value corresponding to the first sliding window. The image feature value corresponding to the (N+1)th sliding window;

[0110] The obtained OCR text may contain repeated characters. The CTC algorithm is used to correct the repeated characters obtained by the LSTM algorithm. The CTC algorithm is a machine learning method for processing sequence data. It is particularly suitable for tasks such as natural language processing, speech recognition and handwriting recognition. Its main application is to map the input sequence to the output sequence when the alignment relationship between the input and output is unknown or not completely clear.

[0111] The main steps of the CTC algorithm are as follows: Input symbol probability distribution: Given an input sequence, it is transformed into a symbol probability distribution at each time step through a neural network or other model, including blank and real symbols. Modeling: The CTC model learns how to distribute blank and real symbols through a neural network to obtain the alignment relationship between the input sequence and the output sequence. This typically involves some recurrent neural networks and fully connected layers. Decoding: Once training is complete, the CTC model can be used to decode the input sequence, mapping it to the output sequence. During decoding, different strategies can be used, such as greedy decoding, beam search, etc., to obtain the final output sequence.

[0112]

[0113] The characters obtained from decoding, For connection-time classification algorithms, The text after removing duplicate characters is obtained; after obtaining the predicted OCR text, the cross-entropy loss is calculated for each character.

[0114]

[0115] It is cross-entropy loss. It is the standard probability that the j-th word in the i-th sample belongs to the c-th category. is the predicted probability that the j-th word in the i-th sample belongs to the c-th category. log() refers to the logarithmic operation. We predefine the length of the OCR text to be 20. OCR text exceeding 20 will discard the extra part, and text less than 20 will be padded with empty characters to 20.

[0116] Step ①_6: Train the local optimal object encoder. For example... Figure 4As shown, a sliding window is used to traverse the entire image from left to right and from top to bottom. Unlike OCR encoders, the dimensions of objects are generally similar, so the sliding window size is set to 3×3, 6×6, and 9×9. For each image in the sliding window, the following operations are performed sequentially: first convolution, second convolution, third convolution, fourth convolution, fifth convolution, first pooling, first fully connected layer, and second fully connected layer, to obtain object features. These operations can be collectively referred to as ResNet operations. ResNet operations use residual functions; the network learns the mapping from the input to the residual, and then adds the residual to the input to produce the output. This method makes it easier for the network to learn the identity mapping because if the identity mapping is optimal, the network can learn a residual close to zero.

[0117]

[0118] in, Let be the image features of the i-th sliding window; calculate the cross-entropy loss between the predicted and true values ​​using the cross-entropy loss function.

[0119]

[0120] in, It is the cross-entropy loss between the predicted and the actual values. It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category.

[0121] Step ①_7: Train the local optimum problem encoder. For example... Figure 5 As shown, the question features are obtained using an embedding layer and three encoding layers. First, the question is expanded to a length of 20 words; if insufficient, it is padded with empty words.

[0122] Then, each word in the question is mapped to a sequence number in the vocabulary list. Subsequently, an embedding layer is used to convert each word into a 768-dimensional feature. Finally, three BERT encoding layers are used to extract the question features.

[0123]

[0124]

[0125]

[0126] It is the input text. It is a mapping operation. It is the mapped text ID. It is an embedding operation. It is an embedded feature. Language model layer It is the third layer of language features. equal .

[0127] The cross-entropy loss function is used to calculate the cross-entropy loss between the predicted and the true values.

[0128]

[0129] in, It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category.

[0130] Step ①_8: Use contrastive learning to jointly train multiple encoders. First, extract the features of each branch using the four pre-trained encoders, denoted as... , , , .

[0131] Then, any two branches are randomly selected for comparative learning. The samples for comparative learning are the true and false relation pairs constructed in step ①_3. The standard value of a true relation pair is 1, and the standard value of a false relation pair is 0. By combining branches pairwise, a total of 6 combinations can be generated.

[0132] Taking the image branch and question branch as examples, assume the image branch has I samples and the question branch has J samples. Calculate the comparative features between the i-th image and the j-th question. Here, the cosine angle is used to represent the degree of separation between features. A cosine of 0 indicates that the two features are uncorrelated, and a cosine of 1 indicates that the two features are positively correlated. This is consistent with the standard value meaning of the relation pairs we constructed.

[0133]

[0134] The loss value for contrastive learning is calculated using the binary cross-entropy loss function.

[0135] Since there are 6 different combinations of branches, the loss value of each combination is multiplied by 1 / 6 to balance the loss among multiple combinations.

[0136]

[0137] Step ①_9: Construct a multimodal Transformer decoding layer to obtain the final answer. The above steps construct a multimodal Transformer encoding layer, whose output is a mixture of features. When constructing the decoding layer, the feature weights of the encoding layer are frozen, and only the feature weights of the decoding layer are adjusted.

[0138] We obtain the answer using a self-decoding approach. Specifically, the input to the decoding layer is preceded by a start character <\s> to indicate the start of prediction. The first character is predicted using <\s>, then the second character is predicted using the first character, and so on, until the <\s> character is predicted. The final <\s> character indicates the end of prediction.

[0139] Each word goes through the same embedding layer, self-attention layer, and feedforward neural network layer as described above to obtain word features. The word then undergoes a Softmax operation to obtain the probability of each character.

[0140] The cross-entropy loss function is used to calculate the cross-entropy loss value between the predicted word probability and the actual word.

[0141]

[0142] Step ①_10: Repeat the training calculation for m iterations to obtain m neural network classification training models and loss function values;

[0143] Step ①_11: Select the weight vector and bias term of the convolutional neural network model corresponding to the smallest value among the loss function values, as the optimal weight vector and optimal bias term;

[0144] The specific steps of the testing phase process are as follows:

[0145] Step ②_1: Construct prediction samples using samples that were not used in training. This includes images and question samples.

[0146] Step ②_2: Calculate the score of each word using the trained optimal neural network, and take the word with the highest score as the answer.

[0147] Step ②_3: Calculate the accuracy rate by comparing the predicted answer with the actual answer. Prediction complete.

[0148] Example 2

[0149] Reference Figure 6 As an embodiment of the present invention, a system for a text visual question answering method based on multi-source interaction is provided, characterized in that it includes a data processing module, an encoder training module, a multimodal contrastive learning module, a decoding prediction module, and a model optimization and evaluation module;

[0150] The data processing module collects data information from the dataset, performs data cleaning and preprocessing, constructs pseudo-relation pairs, and provides the cleaned and preprocessed data to the encoder training module.

[0151] The encoder training module trains the image encoder, OCR encoder, question encoder, and object encoder respectively using image category, OCR question, question context, and object category, extracts their respective features, and transmits the trained encoders to the multimodal contrast learning module.

[0152] The multimodal contrastive learning module performs multimodal contrastive learning between true relation pairs and false relation pairs to align the semantic relations of the multimodal encoder, and then transmits the aligned semantic relations of the multimodal encoder to the decoding and prediction module.

[0153] The decoding and prediction module integrates a multimodal encoder and uses iterative decoding to predict the answer;

[0154] The model optimization and evaluation module is based on the predicted answer, corrects the encoder and decoder parameters, obtains the optimal model, constructs prediction samples using samples that were not used in training, calculates the accuracy, and completes the prediction.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0157] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0158] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0159] Example 3

[0160] Reference Figure 7 This is another embodiment of the present invention. In this embodiment, in order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculations and simulation experiments. This embodiment conducts experiments on both existing conventional methods and the method of this embodiment.

[0161] The effectiveness of this invention was evaluated using accuracy metrics. Experiments were conducted on the TextVQA and STVQA datasets, and the resulting accuracy rates are shown in Table 1. The method of this invention achieved an accuracy of 45.40% on the TextVQA dataset and 42% on the STVQA dataset. Compared to previous methods, this invention achieves higher accuracy and faster response time.

[0162] Table 1. Accuracy of the neural network obtained using the method of this invention on the TextVQA and STVQA datasets.

[0163]

[0164] The prediction speeds on TextVQA and STVQA are shown in Table 2. The unit of prediction speed is samples per second, i.e., the number of samples predicted per second.

[0165] Table 2. Prediction speed of the neural network obtained using the method of this invention on the TextVQA and STVQA datasets.

[0166]

[0167] This method has significantly improved both accuracy and speed, such as Figure 6 As shown in the figure, the dot in the upper right corner represents the method of this invention, and the dot in the lower left corner represents the conventional method. The horizontal axis represents the prediction accuracy, and the vertical axis represents the prediction speed. As can be seen from the figure, both the prediction accuracy and the prediction speed have been significantly improved.

[0168] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text-based visual question answering method based on multi-source interaction, characterized in that, include: Collect data information from the dataset and construct pseudo-relation pairs; The image encoder, OCR encoder, question encoder, and object encoder are trained using image category, OCR question, question context, and object category, respectively. By performing multimodal contrastive learning on true relation pairs and false relation pairs, the semantic relations of the multimodal encoder are aligned. Integrate a multimodal encoder and use iterative decoding to predict the answer; Based on the predicted answer, the encoder / decoder parameters are adjusted to obtain the optimal model; Training a locally optimal image encoder involves using annotation tools to extract OCR text and object category labels from an image, targeting the first... Extract from the images. indivual text , in ;extract Image categories ,in ; Training the encoder involves cropping each image from 256×256 into 32 8×8 partial images, expressed as follows: in, For the entirety of a certain image, The first part of an image. The second part of an image. This is the thirty-second part of a certain image; Each section of images Each image has 3 channels, resulting in 3 × 8 × 8 = 192 pixel values. Each image is unfolded into a one-dimensional array, forming 1 × 192 input data. The entire image has 32 × 192 input data. A 1 × 192 vector with all zero values ​​is defined and concatenated with the unfolded 32 × 192 image data to form the final image input data. The expression is: in, Information about the unfolded image This is the first part of the information in the unfolded overall image. This is the thirty-second section of information in the unfolded overall image. A 1×192 vector (0, 0, ..., 0) of all zeros, with a length of 192, represents the overall features of the image; Input information The total dimension is 33×192. After passing through a linear layer and an activation function layer, the data dimension is expanded to 768 dimensions, as expressed in the following expression: in, Activation function For linear computation operations; Based on the image segmentation process from left to right and top to bottom, each image segment is assigned a sequence number from 0 to 31. An embedding layer converts each image sequence number into a 768-dimensional sequence feature. This sequence feature is then added to the image feature and fed into three consecutive Transformer layers. Each Transformer layer consists of a multi-level self-attention layer and a feedforward neural network layer. The final image layer extracts the overall image features. These features are then processed by a classification layer to predict the image's category. The cross-entropy loss function is used to output the image encoder's prediction, expressed as: in, It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category; Training a locally optimal OCR encoder involves using a sliding window to traverse the entire image from left to right and from top to bottom. The initial size of the sliding window is set to three different sizes, each with a different length and width. Each window is used to traverse the entire image, and the pixels within each window are processed through a convolutional layer and a pooling layer to obtain image features, expressed as: in, Refers to the image feature value corresponding to the i-th sliding window. For pooling operations, For convolution operations, The pixel value of the image in the i-th sliding window; The features of all windows are input into the LSTM encoder and decoder network, and the OCR text of each window is obtained by decoding. The expression is: in, For Long Short-Term Memory (LSTM) module operation, The image feature value corresponding to the first sliding window. The image feature value corresponding to the (N+1)th sliding window; The CTC algorithm is used to correct duplicate characters in OCR text obtained from the LSTM algorithm. The expression is: in, The characters obtained from decoding, For connection-time classification algorithms, This is the text after removing duplicate characters; in, It is the sample number. It is the first word One letter; Set the length of the OCR text, discard OCR text content that exceeds the preset length, and use empty words to fill in OCR text content that is shorter than the preset length; Training a local optimal object encoder involves setting up three sliding windows of different sizes, but each sliding window has the same length and width, and then performing ResNet operations. The ResNet operation involves performing a series of convolutions (first, second, third, fourth, and fifth), pooling, fully connected layers, and a second fully connected layer) sequentially on each image within a sliding window to obtain object features. The expression is as follows: in, For residual network algorithms, The image features of the i-th sliding window; The cross-entropy loss function is used to calculate the cross-entropy loss between the predicted and actual values. The expression is as follows: in, It is the sample size. It is the number of categories. It is the c-th category. It is the standard probability that the i-th sample belongs to the c-th class. Let be the predicted probability that the i-th sample belongs to the c-th category; Training a locally optimal question encoder involves using one embedding layer and three encoding layers to obtain question features. The question is expanded to a preset text length, and empty words are used to pad OCR text content shorter than the preset length. Each word in the question is mapped to a sequence number in a vocabulary list. The embedding layer converts each word into a 768-dimensional feature, and three BERT encoding layers are used to extract the question features. The expression is as follows: in, It is the input text. It is a mapping operation. It is the mapped text ID. It is an embedding operation. It is an embedded feature. Language model layer, It is the third layer of language features; Features of each branch are extracted using four pre-trained encoders. 、 、 、 Arbitrarily select two branches for comparative learning to construct true and false relation pairs. The standard value of a true relation pair is 1, and the standard value of a false relation pair is 0. Pair the branches to generate 6 combinations. The image branch has I samples, and the question branch has J samples. Calculate the contrast features between the i-th image and the j-th question. Use the cosine angle to represent the degree of separation between the features: a cosine of 0 indicates no correlation between the two features, and a cosine of 1 indicates a positive correlation. The expression is: in, It is the inverse cosine function. It is the probability score that the i-th image sample and the j-th question sample are related; The loss value for contrastive learning is calculated using the binary cross-entropy loss function, and the loss across multiple combinations is balanced. The expression is: The branches are paired to generate 6 combinations. The mean is calculated, and the training calculation is repeated for m iterations to obtain m neural network classification training models and loss function values. The weight vector and bias term of the neural network classification training model corresponding to the smallest value among the loss function values ​​are selected as the optimal weight vector and optimal bias term to obtain the optimal model. Construct prediction samples using samples not used in training, calculate the score for each word using the trained optimal neural network, take the word with the highest score as the answer, calculate the accuracy by comparing the predicted answer with the actual answer, and the prediction is complete.

2. The text-based visual question answering method based on multi-source interaction as described in claim 1, characterized in that: Select from the TextVQA dataset The first original image, will be the first The original color image is denoted as , , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and These represent the width and height of the image, respectively. This indicates the position in the original color image. The pixel values ​​of each pixel are used to generate transformed images through data augmentation methods such as horizontal flipping, translation, random channel sorting, and cropping. All images are cropped to a fixed size, and the transformed images corresponding to all the original images are then used to form a training set. The number of transformed images is [number missing]. The first training set A color image is denoted as ,in , It is a positive integer. , The horizontal and vertical coordinates in the image are respectively and point, , , and Let represent the width and height of the image, respectively. Select M question-answer pairs from the TextVQA dataset, and denote the m-th question as . Let the m-th answer be denoted as Where m and M are positive integers, .

3. A system employing the text visual question answering method based on multi-source interaction as described in any one of claims 1 to 2, characterized in that: It includes a data processing module, an encoder training module, a multimodal contrastive learning module, a decoding and prediction module, and a model optimization and evaluation module; The data processing module collects data information from the dataset, performs data cleaning and preprocessing, constructs pseudo-relation pairs, and provides the cleaned and preprocessed data to the encoder training module. The encoder training module trains the image encoder, OCR encoder, question encoder, and object encoder respectively using image category, OCR question, question context, and object category, extracts their respective features, and transmits the trained encoders to the multimodal contrast learning module. The multimodal contrastive learning module performs multimodal contrastive learning between true relation pairs and false relation pairs to align the semantic relations of the multimodal encoder, and then transmits the aligned semantic relations of the multimodal encoder to the decoding and prediction module. The decoding and prediction module integrates a multimodal encoder and uses iterative decoding to predict the answer; The model optimization and evaluation module is based on the predicted answer, corrects the encoder and decoder parameters, obtains the optimal model, constructs prediction samples using samples that were not used in training, calculates the accuracy, and completes the prediction.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the text visual question answering method based on multi-source interaction as described in any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the text visual question answering method based on multi-source interaction as described in any one of claims 1 to 2.

Images