A visual question answering processing method based on memory network and convolution enhancement

Abstract

The application provides a visual question answering processing method based on a memory network and convolution enhancement, belongs to the cross-modal task technical field combining computer vision and natural language processing, and generates memory supplement features of an image from a global perspective by using historical question information related to the image through the memory network, so that complementary information in historical questions corresponding to the same image can be more effectively utilized, and more comprehensive and accurate historical memories are provided for a model. Secondly, convolution enhancement is applied to further extract local key information on question-guided image features, and then the image memory supplement features are reweighted and fused, so that the model pays more attention to the image part most related to the current question when answering the question, the complementary information of the historical question is more accurately extracted from a local perspective, and the accuracy of the visual question answering model is improved.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal applications combining computer vision and natural language processing, and specifically relates to a visual question answering method based on memory-enhanced interaction. Background Technology

[0002] Visual Question Answering (VQA) is a challenging emerging task in the field of artificial intelligence in recent years. It requires models to understand image content and answer related questions based on the images. The emergence of VQA is mainly due to the growing demand for natural language processing and image understanding, and the potential of combining these two approaches. Early models in visual question answering tasks typically extracted global features from the image and question directly as input. However, visual and textual information contains a significant amount of noise, affecting the model's judgment of the answer. To address this problem, researchers began exploring the possibility of introducing attention mechanisms into VQA models, a trend that evolved from initial visual attention to later co-attention mechanisms. Attention mechanisms have achieved significant success in visual question answering tasks, but they also have some limitations.

[0003] First, existing attention-based visual question answering models primarily rely on the fusion of single image-text pairs to generate accurate answers. However, in practical applications, visual question answering tasks involve a wide variety of questions, and the same image may correspond to multiple different visual questions. These questions describe the image content from different perspectives, and the information provided about the environment or objects can complement or refine the answer; this interactive and collaborative information is called complementary information. Although existing improved stacked attention models possess some long-term memory capabilities and can retain a certain degree of image association information, their memory capacity remains insufficient when processing large-scale datasets, making it difficult to effectively utilize the complementary information from multiple visual questions corresponding to the same image.

[0004] Secondly, self-attention mechanisms generate weight distributions by calculating the correlation between each position in the input sequence and all other positions, thereby integrating global information. In this process, the weight of each position depends not only on the features of the current position but also on the features of all other positions in the sequence. This allows self-attention mechanisms to capture global information without being limited by distance, thus enabling the modeling of long-range dependencies. However, because it emphasizes all positions with similar levels of importance, its ability to capture local information is not as good as convolutional neural networks. Convolutional neural networks, when processing sequence data, extract features within a local window through convolution operations and then gradually downsample through operations such as pooling, which can better capture local patterns and local dependencies of features. Summary of the Invention

[0005] To address the problems in existing technologies, this invention provides a visual question answering method based on memory networks and convolutional enhancement. On one hand, it incrementally stores historical question information related to specific images through memory networks and generates memory-enhanced features of the image using the memory network, thereby enabling more effective use of complementary information in historical questions corresponding to the same image. On the other hand, it uses a self-attention mechanism to obtain global features of the image, and applies convolutional operations to further extract local key information of the image, obtaining convolutionally enhanced image features. Finally, it performs reweighted fusion with the memory-enhanced features of the image, enabling the model to extract complementary information from visual questions more accurately and improving the accuracy of the visual question answering model.

[0006] To address the above technical problems, this invention provides the following technical solution: a visual question-answering processing method based on memory networks and convolutional enhancement, comprising the following steps:

[0007] S1. Preprocess the visual question-answering dataset to obtain the question corresponding to the same image.

[0008] S2. For the same image and the corresponding question, this paper performs feature extraction separately to obtain the initial features of the image and the initial features of the question.

[0009] S3. Using the initial features of the image and the initial features of the question as input, and the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement as output, construct a collaborative attention layer to obtain the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement.

[0010] S4. Using the global features of the problem, the global features of the image, and the problem-guided features of the image after convolution enhancement as input, and the memory interaction features of the image as output, a memory interaction layer is constructed to obtain the memory interaction features of the image containing complementary information.

[0011] S5. Using the global features of the problem obtained in step S3, the problem-guided features of the image after convolutional enhancement, and the memory interaction features of the image obtained in step S4 as inputs, and the image-text fusion features as outputs, construct a feature fusion layer, perform feature fusion, and obtain image-text fusion features.

[0012] S6. Using image-text fusion features as input and predicted question answers as output, construct a classification layer to obtain the image prediction answer.

[0013] Furthermore, the aforementioned step S1 includes the following sub-steps:

[0014] S101. Based on the image IDs in the visual question-answering dataset, cluster images with the same ID and their corresponding question texts to obtain the clustered data.

[0015] S102. Load the clustered data obtained in step S101 according to the preset batch. For any batch of data, first sort it randomly, and then group the image-question text pairs according to the ID to obtain the question-text pairs corresponding to the same image.

[0016] Furthermore, in step S2 above, the initial image features of the image are obtained as follows:

[0017] The Faster R-CNN object detector is used to extract features from each image. A confidence threshold is set for the probability of detecting objects in the image, and the number of detected objects is p. The initial features of the image are: Represents p×d v A matrix of dimension d v This represents the feature dimension of each object.

[0018] Furthermore, in step S2 above, the initial characteristics of the problem are obtained as follows:

[0019] S201. Divide the question text into word form, and fill or truncate the number of words contained in each question to the same length n;

[0020] S202. Using a word embedding algorithm pre-trained on a large-scale corpus, each word in the problem is further transformed into a vector, resulting in a word sequence of size n×300.

[0021] S203, through an output d q A dimensional Long Short-Term Memory network is used to obtain the initial features of the problem: Represents n×d q A matrix of dimensionality.

[0022] Furthermore, in the aforementioned step S3, the collaborative attention layer includes l sequentially connected stacked self-attention units and m sequentially connected convolutionally enhanced guided attention units. The stacked self-attention units and the convolutionally enhanced guided attention units are deeply cascaded in an encoder-decoder structure. The first stacked self-attention unit and the first convolutionally enhanced guided attention unit serve as the first input and second input of the collaborative attention layer, respectively. Each convolutionally enhanced guided attention unit includes a sequentially connected first multi-head self-attention-residual connection and normalization module, a second multi-head self-attention-residual connection and normalization module, a convolutionally enhanced FFN module Conv-ffn, and a residual connection and normalization module.

[0023] The first stacked self-attention unit is used to receive the initial features of the problem output from the feature extraction layer, and the first convolutionally enhanced guided attention unit is used to receive the initial features of the image output from the feature extraction layer; each stacked self-attention unit superimposes the features output by the previous stacked self-attention unit, and the last stacked self-attention unit serves as the first output of the collaborative attention layer, used to output the global features of the problem.

[0024] The last stacked self-attention unit outputs the global features of the problem to the first convolutionally enhanced guided attention unit. The first convolutionally enhanced guided attention unit convolves and enhances the initial features of the image and the global features of the problem, and then outputs them to the convolutionally enhanced guided attention units that are sequentially connected after the first convolutionally enhanced guided attention unit, thus obtaining the global features of the image and the problem-guided features of the image after convolution enhancement.

[0025] Furthermore, in step S3 above, each stacked self-attention unit is configured to perform a two-stage operation to obtain the global features of the problem, specifically:

[0026] The first stage is SA phase1 The first stage includes a multi-head self-attention stage, a residual connection stage, and a normalization stage; the second stage is SA. phase2 This includes a feedforward neural network, a residual connection, and a normalization process, as shown in the following formula:

[0027] SA_Q o =SA_Q i +norm(MSA(SA_Q i )) (1)

[0028] SA_Q s =SA_Q o +norm(FFN(SA_Q o (2)

[0029] Among them, SA_Q i SA_Q represents the problem characteristics processed by stacked self-attention units. o SA_Q represents the problem feature obtained after the first stage operation of stacked self-attention units. s This represents the final output feature of a single-layer stacked self-attention unit after the second stage operation of the stacked self-attention units; + indicates residual connection operation, norm() indicates normalization, MSA() indicates multi-head attention operation, and FFN() indicates nonlinear transformation of the feature.

[0030] SA_Q sThe final self-attention feature, namely the global feature SA_Q of the problem, is obtained by stacking multiple layers of self-attention units, and is calculated as shown in formula (3):

[0031]

[0032] Where l represents the number of layers of the self-attention units stacked in the multi-layer stack, l∈[1,6], This represents the first stage operation of the self-attention units stacked in the l-th layer. This represents the second-stage operation of the self-attention units stacked at layer l. For the self-attention units stacked at layer l, SA_Q l-1 SA_Q represents the input of this layer. l This is represented as the output of this layer, and is recursively fed into the (l+1)th layer of stacked self-attention units as input. The input SA_Q to the first layer of stacked self-attention units... 0 This is the initial characteristic Q of the problem.

[0033] Furthermore, in step S3 above, the question guidance features of the convolutionally enhanced image are obtained by stacking m identical convolutionally enhanced guiding attention units. Each convolutionally enhanced guiding attention unit is configured to perform two-stage operations to obtain the question guidance features of the convolutionally enhanced image, specifically:

[0034] The first stage is GCA phase1 The initial features V of the image are processed through a multi-head self-attention stage, a residual connection stage, and a normalization stage to obtain the global features GCA_V of the image. global Then GCA_V global The global features SA_Q of the problem are then processed again through a multi-head self-attention stage, a residual connection stage, and a normalization stage to obtain the problem-guided image features GCA_V. guided ;

[0035] The second phase is GCA phase2 GCA_V guided The input is a convolution-enhanced FFN module (Conv-ffn), a residual connection and normalization module, to obtain the local features GCA_V of the image. local ;

[0036] Local features of an image GCA_V local The calculation process is shown in formulas (4) to (6):

[0037] GCA_V local1 =gel u(Conv1(GCA_V guided (4)

[0038] GCA_Vlocal2 =DWconv(GCA_V local1 (5)

[0039] GCA_V local =gelu(Conv2(GCA_V local2 (6)

[0040] The image features GCA_V are obtained after the first and second convolutional operations of the FFN module Conv-ffn, which enhances convolution. local1 With GCA_V local2 GCA_V local This represents the local features of the image output by the Conv-ffn layer of the Conv-ffn module after the third convolutional operation of the Conv-ffn module. Conv1 and Conv2 are regular 2D convolutions, and DWconv is a depthwise separable convolution. The operation is divided into two steps: depthwise convolution and pointwise convolution. Finally, the local features of the image are GCA_V. local Image features GCA_V guided by the question guided Residual connections and layer normalization are performed to obtain more comprehensive guidance attention features, calculated as shown in formula (7):

[0041] GCA_V t =GCA_V guided +norm(GCA_V local (7)

[0042] Among them, GCA_V t This represents the question-guided feature of the convolutionally enhanced image obtained after processing by a single-layer convolutional enhanced guiding attention unit. This feature integrates local and global information of the image under question guidance.

[0043] GCA_V t The problem-guided feature GCA_V of the final convolutionally enhanced image is obtained by stacking multiple layers of convolutionally enhanced guided attention units. The calculation process is as shown in formula (8):

[0044]

[0045] Where m represents the number of layers of stacked guided attention units for convolutional enhancement, m∈[1,6], This represents the first stage operation of the guided attention unit in the m-th convolutional layer. This represents the second-stage operation of the guided attention unit with convolutional enhancement at layer m; for the guided attention unit with convolutional enhancement at layer m, GCA_V m-1 GCA_V represents the input of this layer. mThis is represented as the output of this layer being recursively fed into the (m+1)th layer's convolutionally enhanced guided attention unit as input. The input GCA_V to the first layer's convolutionally enhanced guided attention unit... 0 This represents the initial feature V of the image, and the global feature GCA_V of the image. global It is updated layer by layer through multiple layers.

[0046] Furthermore, in the aforementioned step S4, constructing the memory interaction layer includes the generation of input features, the storage of general memory, the calculation of attention, and the output of memory interaction features, specifically including the following steps:

[0047] The following sub-steps:

[0048] S401. Generation of input features: Obtain the feature vector SA_Q corresponding to the current problem based on the image ID. id Generate a tuple (id, SA_Q) id ) as input to the memory interaction layer;

[0049] S402. Storage of General Memory: General memory M uses key-value pairs for storage, with image id as the key and the concatenated feature vector of the corresponding historical question as the value. Key-value storage is incremental; for a pair (id, SA_Q) in the input features... id If the image ID is not found in the keys of the general memory M, a new entry is added with the image ID as the key and its value is the feature vector SA_Q of the current problem. id If the id is found in the key of the general memory M, then first extract its corresponding value SA_Q_M. id Then, it is combined with the feature vector SA_Q of the current problem. id After splicing, the original value is updated, and all historical information corresponding to the image is remembered incrementally;

[0050] S403, Attention Calculation: Utilizes the historical problem features SA_Q_M stored in the universal memory M, and the global image features GCA_V generated during the computation of the convolution-enhanced guided attention unit. global The memory supplementation feature ATT_V of the image is obtained after attention operation, and the calculation process is as shown in formula (9):

[0051] ATT_V=attention(SA_Q_M,SA_Q_M,GCA_V global (9)

[0052] Where attention is used for attention calculation;

[0053] S404. Output of Memory Interaction Features: The memory interaction features ATT_V of the image are obtained by performing a reweighted fusion operation on the problem-guided features GCA_V and the memory supplement features ATT_V of the image after convolution enhancement. R The calculation process is as shown in formulas (10) to (12):

[0054] a_v = softmax(GCA_V) (10)

[0055] ATT_V rw =fc(a_v*ATT_V) (11)

[0056] ATT_V R =concat(ATT_V rw ,fc(GCA_V)) (12)

[0057] Where a_v is the probability distribution obtained by normalizing GCA_V, softmax is the normalized activation function, fc is the fully connected layer, and concat is the vector concatenation operation; a_v is obtained by normalizing GCA_V using softmax, and ATT_V is obtained by reweighting ATT_V using a_v. rw Finally, GCA_V and ATT_V rw The final image memory interaction feature ATT_V is obtained by concatenating the data after dimensionality transformation through a fully connected layer. R .

[0058] Furthermore, in step S5 mentioned above, a feature fusion layer is constructed to perform feature fusion. Specifically, this involves processing the output ATT_V of the memory interaction layer. R The image-text fusion feature JOINT is obtained by performing a reweighted fusion operation again, which optimizes the image features at a deeper level and accurately extracts complementary information in visual problems. The calculation process is as shown in formulas (13) to (16):

[0059] joint=GCA_V*SA_Q (13)

[0060] a_j = softmax(joint) (14)

[0061] V J =fc(a_j*ATT_V R (15)

[0062] JOINT = concat(V J ,f(joint)) (16)

[0063] Where joint is the initial image-text fusion feature, a_j is the probability distribution obtained after normalizing joint, and V JFor a_j to ATT_V R Image features obtained after reweighting.

[0064] Furthermore, step S6 described above specifically involves: inputting the image-text fusion feature JOINT into a network composed of multiple linear layers, and finally controlling the result between 0 and 1 using the sigmoid function as the probability of each candidate answer. The classification calculation process is as shown in formula (17):

[0065] S=sigmoid(linear(JOINT)) (17)

[0066] Where sigmoid is a non-linear activation function, S∈R A R predicts the answer for the model. A Let S be the mappable answer space, A be the number of pre-defined answers in the training set, and linear represent the transformation of features through multiple linear layers. The answer with the highest probability in S is selected as the model's predicted answer, and the binary cross-entropy function is used as the loss function to train the multi-classifier.

[0067] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0068] 1. Initial image features are optimized using convolutionally enhanced guided attention units. Self-attention and guided attention mechanisms are employed to obtain question-guided global image features. These features are then fed into a convolutionally enhanced feedforward neural network layer (Conv-ffn) to further extract key local information, resulting in question-guided features of the convolutionally enhanced image. Conv-ffn enhances the model's ability to capture local features and spatial structure, and also improves its generalization ability, thereby accurately extracting and generating more accurate and representative image features for specific questions from historical information provided by visual problems.

[0069] 2. This method incrementally stores historical question information associated with images using a memory network, and generates memory-complementary features of the image through attention computation with captured global image features. These features are then reweighted and fused with question-guided features of the convolutionally enhanced image to generate memory-interactive features. This approach leverages the latent connections between questions and integrates complementary information from them, providing the model with a more comprehensive and accurate historical memory. Attached Figure Description

[0070] Figure 1 This is a flowchart of the method of the present invention.

[0071] Figure 2 This is a model framework diagram of the present invention. Detailed Implementation

[0072] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0073] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0074] refer to Figure 1 This invention provides a visual question-answering processing method based on memory networks and convolutional enhancement, the steps of which are as follows:

[0075] S1. Preprocess the visual question-answering dataset to obtain the questions corresponding to the same image. The specific implementation is as follows:

[0076] refer to Figure 1 The preprocessing layer shown has a batch size of 64. Each batch of data is organized as a dictionary, containing image ID, image features, question text, and answer. First, image-question text pairs are sorted according to their image IDs in the original dataset, ensuring that pairs with the same image ID are grouped together. When loading the dataset, the Fisher-Yates Shuffle algorithm is first used to shuffle all the data. Then, image-question text pairs are grouped by image ID, ensuring that they are input into the model as groups, maximizing the loading of visual questions for the same image together within limited memory. Finally, each batch of data is sequentially input into the model for training.

[0077] S2. For the same image and the corresponding question, this paper performs feature extraction separately to obtain the initial features of the image and the initial features of the question.

[0078] like Figure 1 As shown in the feature extraction layer, the initial features of the image are obtained by extracting them from the original image using the object detector Faster R-CNN (with ResNet-101 as the backbone), which is pre-trained on the Visual Genome (VG) dataset. A confidence threshold is set for the probability of objects detected in the image, and the number of detected objects is obtained as p. The initial features of the image are: Represents p×d v A matrix of dimension d vThis represents the feature dimension of each object; in the example, the number of objects contained in each image feature is p∈[10, 100], and the feature dimension is 2048. To facilitate batch calculation, the number of objects contained in the image features is uniformly padded or truncated to the same number 100. Finally, the image features are represented by a matrix V∈R. 100×2048 R 100×2048 This represents a matrix of 100×2048 dimensions.

[0079] The initial characteristics of the problem are obtained through the following steps:

[0080] S201. Divide the question text into word form, and pad or truncate the number of words in each question to the same length n. In this example, n = 14.

[0081] S202. Using a 300-dimensional GloVe word embedding algorithm pre-trained on a large-scale corpus, each word in the problem is further transformed into a vector, resulting in a word sequence of size n×300, which is 14×300 in this case.

[0082] S203. Using a Long Short-Term Memory network with output dimension dq, the initial features of the problem are obtained as follows: Represents n×d q A matrix of dimensions. In this example, the output is a Long Short-Term Memory (LSTM) network with a dimension of 1024, outputting the problem features Q∈R. 14×1024 R 14×1024 This represents a matrix of 14×1024 dimensions.

[0083] S3. Using the initial features of the image and the initial features of the question as input, and the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement as output, a collaborative attention layer is constructed to obtain the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement. (Reference) Figure 2The collaborative attention layer in the model comprises l sequentially connected stacked self-attention units (SA) and m sequentially connected guided-convolution-attention (GCA) units. The V and Q obtained in step S2 learn a deep interaction between the two modal information through the collaborative attention layer. The SA and GCA units are deeply cascaded in an encoder-decoder structure. The first stacked self-attention unit and the first convolutionally enhanced guided attention unit serve as the first and second inputs of the collaborative attention layer, respectively. Each convolutionally enhanced guided attention unit includes a sequentially connected first multi-head self-attention-residual connection and normalization module, a second multi-head self-attention-residual connection and normalization module, a convolutionally enhanced FFN module (Conv-ffn), and a residual connection and normalization module.

[0084] The first stacked self-attention unit is used to receive the initial features of the problem output from the feature extraction layer, and the first convolutionally enhanced guided attention unit is used to receive the initial features of the image output from the feature extraction layer; each stacked self-attention unit superimposes the features output by the previous stacked self-attention unit, and the last stacked self-attention unit serves as the first output of the collaborative attention layer, used to output the global features of the problem.

[0085] The last stacked self-attention unit outputs the global features of the problem to the first convolutionally enhanced guided attention unit. The first convolutionally enhanced guided attention unit convolves and enhances the initial features of the image and the global features of the problem, and then outputs them to the convolutionally enhanced guided attention units that are sequentially connected after the first convolutionally enhanced guided attention unit, thus obtaining the global features of the image and the problem-guided features of the image after convolution enhancement.

[0086] Step S3 includes an encoding stage and a decoding stage. The encoding stage involves each stacked self-attention unit being configured to perform two stages of operations to obtain the global features SA_Q of the problem. Specifically:

[0087] The first stage is SA phase1 It includes a multi-head self-attention (MSA) stage, a residual connection stage (add) stage, and a normalization stage (norm) stage;

[0088] The second stage is SA. phase2 It includes a feedforward network (FFN) stage, a residual connection stage (add), and a normalization stage (norm), as shown in the following formula:

[0089] SA_Q o =SA_Q i +norm(MSA(SA_Q i )) (1)

[0090] SA_Q s =SA_Q o +norm(FFN(SA_Q o (2)

[0091] Among them, SA_Q i SA_Q represents the problem characteristics processed by stacked self-attention units. o SA_Q represents the problem feature obtained after the first stage operation of stacked self-attention units. s This represents the final output feature of a single-layer stacked self-attention unit after the second stage operation of the stacked self-attention units; + indicates residual connection operation, norm() indicates normalization, MSA() indicates multi-head attention operation, and FFN() indicates nonlinear transformation of the feature.

[0092] SA_Q s The final self-attention feature, namely the global feature SA_Q of the problem, is obtained by stacking multiple layers of self-attention units, and is calculated as shown in formula (3):

[0093]

[0094] Where l represents the number of layers of the self-attention units stacked in the multi-layer stack, l∈[1,6], This represents the first stage operation of the self-attention units stacked in the l-th layer. This represents the second-stage operation of the self-attention units stacked at layer l. For the self-attention units stacked at layer l, SA_Q l-1 SA_Q represents the input of this layer. l This is represented as the output of this layer, and is recursively fed into the (l+1)th layer of stacked self-attention units as input. The input SA_Q to the first layer of stacked self-attention units... 0 This is the initial characteristic Q of the problem.

[0095] The decoding stage in step S3 is as follows: Problem guidance features GCA_V of the convolutionally enhanced image are obtained by stacking m identical convolutionally enhanced guiding attention units. Each convolutionally enhanced guiding attention unit is configured to perform two-stage operations, resulting in the following specific problem guidance features of the convolutionally enhanced image:

[0096] The first stage is GCA phase1The initial features V of the image are processed through a multi-head self-attention stage, a residual connection stage, and a normalization stage to obtain the global features GCA_V of the image. global Then GCA_V global The global features SA_Q of the problem are then processed again through a multi-head self-attention (MSA) stage, a residual connection stage (add), and a normalization stage (norm) to obtain the problem-guided image features GCA_V. guided ;

[0097] The second phase is GCA phase2 GCA_V guided The input is a convolution-enhanced FFN module (Conv-ffn), a residual connection and normalization module, to obtain the local features GCA_V of the image. local The convolution-enhanced FFN module Conv-ffn consists of three convolutional layers and a Gelu activation function, used to capture local image features GCA_V. local In Conv-ffn, the first and third layers use conventional convolution operations to perform feature dimensionality enhancement (i.e., Enlarge layer) and downsampling (i.e., Shrink layer) operations, respectively, while the intermediate layers use depthwise separable convolution.

[0098] Local features of an image GCA_V local The calculation process is shown in formulas (4) to (6):

[0099] GCA_V local1 =gel u(Conv1(GCA_V guided (4)

[0100] GCA_V local2 =DWconv(GCA_V local1 (5)

[0101] GCA_V local =gelu(Conv2(GCA_V local2 (6)

[0102] The image features GCA_V are obtained after the first and second convolutional operations of the FFN module Conv-ffn, which enhances convolution. local1 With GCA_V local2 GCA_V localThis represents the local features of the image output by the Conv-ffn layer of the Conv-ffn module after the third convolutional operation of the Conv-ffn module. Conv1 and Conv2 are regular 2D convolutions, and DWconv is a depthwise separable convolution. The operation is divided into two steps: depthwise convolution and pointwise convolution. Finally, the local features of the image are GCA_V. local Image features GCA_V guided by the question guided Residual connections and layer normalization are performed to obtain more comprehensive guidance attention features, calculated as shown in formula (7):

[0103] GCA_V t =GCA_V guided +norm(GCA_V local (7)

[0104] Among them, GCA_V t This represents the question-guided feature of the convolutionally enhanced image obtained after processing by a single-layer convolutional enhanced guiding attention unit. This feature integrates local and global information of the image under question guidance.

[0105] GCA_V t The problem-guided feature GCA_V of the final convolutionally enhanced image is obtained by stacking multiple layers of convolutionally enhanced guided attention units. The calculation process is as shown in formula (8):

[0106]

[0107] Where m represents the number of layers of stacked guided attention units for convolutional enhancement, m∈[1,6], This represents the first stage operation of the guided attention unit in the m-th convolutional layer. This represents the second-stage operation of the guided attention unit with convolutional enhancement at layer m; for the guided attention unit with convolutional enhancement at layer m, GCA_V m-1 GCA_V represents the input of this layer. m This is represented as the output of this layer being recursively fed into the (m+1)th layer's convolutionally enhanced guided attention unit as input. The input GCA_V to the first layer's convolutionally enhanced guided attention unit... 0 This represents the initial feature V of the image, and the global feature GCA_V of the image. global It is updated layer by layer through multiple layers.

[0108] S4. Using the global features of the problem, the global features of the image, and the problem-guided features of the image after convolutional enhancement as input, and the memory interaction features of the image as output, a memory interaction layer is constructed to obtain the memory interaction features of the image containing complementary information. (Reference) Figure 2Specifically, it includes the following steps:

[0109] S401. Generation of Input Features (I): Obtain the feature vector SA_Q corresponding to the current problem based on the image ID. id Generate a tuple (id, SA_Q) id ) as input to the memory interaction layer;

[0110] S402, General Memory Storage (M): Memory M uses key-value pairs for storage, with image id as the key and the concatenated feature vector of the corresponding historical question as the value. Key-value storage is incremental; for a pair (id, SA_Q) in the input features... id If the image ID is not found in the keys of the general memory M, a new entry is added with the image ID as the key and its value is the feature vector SA_Q of the current problem. id If the id is found in the key of the general memory M, then first extract its corresponding value SA_Q_M. id Then, it is combined with the feature vector SA_Q of the current problem. id After splicing, the original value is updated, and all historical information corresponding to the image is remembered incrementally;

[0111] S403, Attention Calculation (A): Using the historical problem features SA_Q_M stored in the universal memory M, and the global image features GCA_V generated during the computation of the convolution-enhanced guided attention unit. global The memory supplementation feature ATT_V of the image is obtained after attention operation, and the calculation process is as shown in formula (9):

[0112] ATT_V=attention(SA_Q_M,SA_Q_M,GCA_V global (9)

[0113] Where attention is used for attention calculation;

[0114] S404. Output of Memory Interaction Feature (R): The memory interaction feature ATT_V of the image is obtained by reweighting and fusing the problem-guided feature GCA_V and the memory supplement feature ATT_V of the image after convolution enhancement. R The calculation process is as shown in formulas (10) to (12):

[0115] a_v = softmax(GCA_V) (10)

[0116] ATT_V rw =fc(a_v*ATT_V) (11)

[0117] ATT_V R=concat(ATT_V rw ,fc(GCA_V)) (12)

[0118] Where a_v is the probability distribution obtained by normalizing GCA_V, softmax is the normalized activation function, fc is the fully connected layer, and concat is the vector concatenation operation; a_v is obtained by normalizing GCA_V using softmax, and ATT_V is obtained by reweighting ATT_V using a_v. rw Finally, GCA_V and ATT_V rw The final image memory interaction feature ATT_V is obtained by concatenating the data after dimensionality transformation through a fully connected layer. R .

[0119] S5. The global features of the problem obtained in step S3, the problem-guided features of the image after convolutional enhancement, and the memory interaction features ATT_V of the image obtained in step S4. R Using the image-text fusion features as input, a feature fusion layer is constructed with the image-text fusion features as output to perform feature fusion, optimize image features at a deeper level, accurately extract complementary information in visual problems, and obtain the image-text fusion feature JOINT. The specific calculation process is shown in formulas (13) to (16):

[0120] joint=GCA_V*SA_Q (13)

[0121] a_j = softmax(joint) (14)

[0122] V J =fc(a_j*ATT_V R (15)

[0123] JOINT = concat(V J fc(joint)) (16)

[0124] Where joint is the initial image-text fusion feature, a_j is the probability distribution obtained after normalizing joint, and V J For a_j to ATT_V R Image features obtained after reweighting.

[0125] S6. Using the image-text fusion feature JOINT as input and the predicted question answer as output, a classification layer is constructed to obtain the image-predicted answer. Specifically, the image-text fusion feature JOINT is input into a network composed of multiple linear layers, and finally the result is controlled between 0 and 1 by the sigmoid function, which serves as the probability of each candidate answer. The classification calculation process is shown in formula (17):

[0126] S=sigmoid(linear(JOINT)) (17)

[0127] Where sigmoid is a non-linear activation function, S∈R A R predicts the answer for the model. A Let S be the mappable answer space, A be the number of most common answers in the training set, and linear represent the features transformed through multiple linear layers. The answer with the highest probability in S is selected as the model's predicted answer, and the binary cross-entropy function (BCE) is used as the loss function to train the multi-classifier.

[0128] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A visual question-answering processing method based on memory networks and convolutional enhancement, characterized in that, Includes the following steps: S1. Preprocess the visual question-answering dataset to obtain the question corresponding to the same image. S2. For the same image and the corresponding question, this paper performs feature extraction separately to obtain the initial features of the image and the initial features of the question. S3. Using the initial features of the image and the initial features of the question as input, and the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement as output, construct a collaborative attention layer to obtain the global features of the question, the global features of the image, and the question-guided features of the image after convolutional enhancement. The collaborative attention layer includes l sequentially connected stacked self-attention units and m sequentially connected convolutionally enhanced guided attention units. The stacked self-attention units and the convolutionally enhanced guided attention units are deeply cascaded in an encoder-decoder structure. The first stacked self-attention unit and the first convolutionally enhanced guided attention unit serve as the first input and second input of the collaborative attention layer, respectively. Each convolutionally enhanced guided attention unit includes a sequentially connected first multi-head self-attention-residual connection and normalization module, a second multi-head self-attention-residual connection and normalization module, and a convolutionally enhanced FFN module Conv-ffn, a residual connection and normalization module. The first stacked self-attention unit is used to receive the initial features of the problem output from the feature extraction layer, and the first convolutionally enhanced guided attention unit is used to receive the initial features of the image output from the feature extraction layer; each stacked self-attention unit superimposes the features output by the previous stacked self-attention unit, and the last stacked self-attention unit serves as the first output of the collaborative attention layer, used to output the global features of the problem. The last stacked self-attention unit outputs the global features of the problem to the first convolutionally enhanced guided attention unit. The first convolutionally enhanced guided attention unit convolves and enhances the initial features of the image and the global features of the problem, and then outputs them to the convolutionally enhanced guided attention units that are sequentially connected after the first convolutionally enhanced guided attention unit, thus obtaining the global features of the image and the problem-guided features of the image after convolution enhancement. S4. Using the global features of the problem, the global features of the image, and the problem-guided features of the image after convolution enhancement as input, and the memory interaction features of the image as output, a memory interaction layer is constructed to obtain the memory interaction features of the image containing complementary information. In step S4, constructing the memory interaction layer includes the generation of input features, the storage of general memory, the calculation of attention, and the output of memory interaction features, specifically including the following steps: S401. Generation of input features: based on the image Obtain the feature vector of the current problem corresponding to this image. Generate a tuple As input to the memory interaction layer; S402, Storage of General Memory: General Memory M uses key-value pairs to store memory, in the form of images. The key is the image itself, and the value is the concatenated feature vector of the corresponding historical question. Key-value pairs are stored incrementally, with each pair representing a given input feature. No results were found in the key of the general memory M. Then add an image The key is an entry whose value is the feature vector of the current problem. If found in the key of general memory M; Then first extract its corresponding value. Then, with the feature vector of the current problem After splicing, the original value is updated, and all historical information corresponding to the image is remembered incrementally; S403, Attention Calculation: Using features of historical problems stored in general memory M. And the global image features generated during the computation of the convolution-enhanced guided attention unit. The memory supplementation features of the image are obtained through attention operations. The calculation process is as follows: ， in, Calculations for attention; S404, Output of memory interaction features: Question-guided features for the convolution-enhanced image. and image memory supplementary features The memory interaction features of the image are obtained by performing a reweighted fusion operation. The calculation process is as follows: ， ， , ， in, To The probability distribution obtained after normalization For normalized activation functions, It is a fully connected layer. This is a vector concatenation operation; via Normalization Later obtained ,use right Reweighting to obtain Finally, and The final image memory interactive features are obtained by concatenating the data after dimensionality transformation through fully connected layers. ; S5. Using the global features of the problem obtained in step S3, the problem-guided features of the image after convolutional enhancement, and the memory interaction features of the image obtained in step S4 as inputs, and the image-text fusion features as outputs, construct a feature fusion layer, perform feature fusion, and obtain image-text fusion features. S6. Using image-text fusion features as input and predicted question answers as output, construct a classification layer to obtain the image prediction answer.

2. The visual question-answering processing method based on memory networks and convolutional enhancement according to claim 1, characterized in that, Step S1 includes the following sub-steps: S101. Based on the images in the visual question answering dataset The system groups images with the same ID and their associated question text to obtain clustered data. S102. Load the clustered data obtained in step S101 according to the preset batches. For any batch of data, first, perform random sorting, and then... Grouping image-question-text pairs yields question-text pairs corresponding to the same image.

3. The visual question-answering method based on memory networks and convolutional enhancement according to claim 1, characterized in that, In step S2, the initial image features are obtained as follows: The Faster R-CNN object detector is used to extract features from each image. A confidence threshold is set for the probability of detecting objects in the image, and the number of detected objects is obtained. The initial features of the image are: , express A matrix of dimension This represents the feature dimension of each object.

4. The visual question-answering method based on memory networks and convolutional enhancement according to claim 1, characterized in that, In step S2, the initial features of the problem are obtained as follows: S201. Divide the question text into word form, and pad or truncate the number of words in each question to the same length. S202. Using a word embedding algorithm pre-trained on a large-scale corpus, each word in the problem is further transformed into a vector, resulting in a vector of size S202. Word sequence, S203, through an output as A dimensional Long Short-Term Memory network is used to obtain the initial features of the problem: , express A matrix of dimensionality.

5. The visual question-answering method based on memory networks and convolutional enhancement according to claim 4, characterized in that, In step S3, each stacked self-attention unit is configured to perform a two-stage operation to obtain the global features of the problem, specifically: The first stage is This includes the multi-head self-attention stage, the residual connection stage, and the normalization process stage; The second stage is It includes a feedforward neural network, a residual connection, and a normalization process, as shown in the following formula: ， ， in, This represents the problem characteristics processed by the self-attention units of the input stack. This represents the problem features obtained after the first stage of operations through stacked self-attention units. This represents the problem features of the final output of a single-layer stacked self-attention unit obtained after the second stage operation of the stacked self-attention units; This indicates a residual join operation. This indicates normalization processing. This indicates a bullish attention-based operation. This indicates a nonlinear transformation of the feature; The final self-attention feature, i.e., the global feature of the problem, is obtained by stacking multiple layers of self-attention units. The calculation is performed using the following formula: ； in, This indicates the number of layers in a multi-layered stack of self-attention units. , Indicates the first The first stage operation of the stacked self-attention units. Indicates the first The second stage operation of the stacked self-attention units, for the first... Stacked self-attention units This is represented as the input of this layer. This is represented as the output of this layer, and is further fed into the next layer recursively. The self-attention units are stacked in layers as input. For the input of the self-attention unit in the first layer of stacking... This constitutes the initial characteristics of the problem. .

6. The visual question-answering method based on memory networks and convolutional enhancement according to claim 5, characterized in that, In step S3, through To obtain the question-guided features of the convolutionally enhanced image, identical convolutionally enhanced guided attention units are stacked. Each convolutionally enhanced guided attention unit is configured to perform a two-stage operation to obtain the question-guided features of the convolutionally enhanced image, specifically: The first stage is Initial features of the image After passing through a multi-head self-attention stage, a residual connection stage, and a normalization stage, the global features of the image are obtained. ;Then With the global characteristics of the problem After passing through a multi-head self-attention stage, a residual connection stage, and a normalization stage, the problem-guided image features are obtained. ; The second stage is ,Will Inputting the convolutionally enhanced FFN module Conv-ffn, residual connections, and normalization module, local features of the image are obtained. ; Local features of an image The calculation process is shown in the following formula: ， ， ， The image features are obtained after the first and second convolutional operations of the Conv-ffn module, which enhances convolution. and , This represents the local features of the final output image of the Conv-ffn layer of the Conv-ffn module after the third convolution operation of the Conv-ffn module. and For regular two-dimensional convolution, For depthwise separable convolution, the operation is divided into two steps: depthwise convolution and pointwise convolution, and finally, the local features of the image are processed. Image features guided by questions By performing residual connections and layer normalization, we obtain more comprehensive guidance attention features, calculated as shown in the following formula: ， in, This represents the question-guided feature of the convolutionally enhanced image obtained after processing by a single-layer convolutional enhanced guiding attention unit. This feature integrates local and global information of the image under question guidance. The final convolutionally enhanced image has question-guided features obtained by stacking multiple layers of convolutional enhanced guiding attention units. The calculation process is as follows: , in, This indicates the number of layers in the stack of guided attention units for convolutional enhancement. , This represents the first stage operation of the guided attention unit in the m-th convolutional layer. Indicates the first The second stage operation of the guided attention unit enhanced by layer convolution; for the first... Layered convolutional enhanced guided attention units, This is represented as the input of this layer. This is represented as the output of this layer, which is then recursively input into the next layer. The first layer of convolutionally enhanced guided attention units is used as input. These are the initial features of the image. Global features of the image It is updated layer by layer through multiple layers.

7. The visual question-answering method based on memory networks and convolutional enhancement according to claim 6, characterized in that, In step S5, a feature fusion layer is constructed to perform feature fusion. Specifically, this involves processing the output of the memory interaction layer. Perform a reweighted fusion operation again to obtain the image-text fusion features. To optimize image features at a deeper level and accurately extract complementary information from visual problems, the calculation process is as follows: ， ， ， ， in, For the initial image-text fusion features, To The probability distribution obtained after normalization for right Image features obtained after reweighting.

8. The visual question-answering method based on memory networks and convolutional enhancement according to claim 7, characterized in that, Step S6 specifically involves: integrating image and text features. The input is a network consisting of multiple linear layers, and finally... The function controls the result to be between 0 and 1, representing the probability of each candidate answer. The classification calculation process is as follows: ， in, It is a non-linear activation function. ∈ To predict the answer for the model, Let A be the space of mappable answers, and A be the number of pre-defined answers in the training set. The features represent those transformed through multiple linear layers, and the values ​​are taken as follows: The answer with the highest probability is the one predicted by the model, and the binary cross-entropy function is used as the loss function to train the multi-classifier.

Images