A visual question answering method based on cross-modal pre-training feature enhancement

Abstract

The present invention discloses a visual question answering method based on cross-modal pre-trained feature enhancement. The present invention models relative spatial relationship features, utilizes semantically aligned local / global visual features and language features, combines a feature enhancement module and a deeply stacked language-side SA module and a visual-side RGA module to obtain richer composite semantic information, and finally outputs comprehensive features through a local / global feature fusion module for predicting answers. This method has achieved significant improvement in visual question answering tasks, surpassing many methods that use traditional object features or grid features as visual features, and even outperforming many models that use the vision-language "pre-training-fine-tuning" paradigm.

Description

Technical Field

[0001] The present invention proposes a visual question answering method based on cross-modal pre-trained feature enhancement (XPFE-VQA). Background Art

[0002] Visual Question Answering (VQA) is a very challenging task in the field of visual-language cross-modality. The visual question answering task aims to give an image and a natural language description question related to the image content as input, requiring the algorithm to understand and reason about the image and the question at the same time, and finally output an accurate answer in the form of natural language. For example, the image content is a train platform, there is a train next to the platform, there are rest tables and chairs on the platform, a man is sitting at the table with a laptop, and a bicycle is parked on the left side of the table. Given questions about the image content, such as "What is this place?", "What color clothes does the man at the table wear?", "What is in the lower left corner of the image?", etc., the model needs to perceive the scene of the image and understand the question, and then infer the corresponding answer. The visual question answering task is an intersection of the fields of computer vision and natural language processing, and has extremely high requirements for algorithm and model capabilities. How to design a model that effectively utilizes the semantic information of images and questions and has the ability to integrate and reason cross-modal information is the key to studying the field of visual question answering.

[0003] In recent years, deep learning-based methods have received great attention from academia and industry, and have produced many excellent network models and various effective training methods. The rich research on single modalities of vision or language supports cross-modal research, making visual-language cross-modal tasks including visual question answering a current hot research direction. In visual question answering algorithms, how to effectively construct features for input images and questions and perform cross-modal information fusion, interaction, and reasoning without using a pre-trained object detection model, only using an end-to-end optimization model based on a deep neural network is a research issue worthy of in-depth exploration.

[0004] In terms of practical application, visual question answering technology has a very wide range of application scenarios. According to the "White Paper on Chinese People with Disabilities in 2018", as of May 2018, the number of disabled people in my country was close to 90 million, of which the number of visually impaired people reached 18 million, equivalent to one visually impaired person for every 80 people in my country. The visual system is the main way for humans to perceive environmental information, and more than 80% of information is obtained through the eyes. At present, the visual information perception and interaction needs of visually impaired people in daily life are difficult to be fully met, which is becoming a problem that needs to be solved urgently. The perception and interaction assistance system for visually impaired people based on visual question answering technology will have great market application value and social benefits.

[0005] In summary, visual question answering is a topic worthy of in-depth research. This patent intends to explore several key points in this task, solve the difficulties and key points of current methods, and form a complete visual question answering system.

[0006] The correct answers to visual question answering tasks come from different annotators, which has a high degree of freedom. The image carriers in natural scenes have diverse themes, and the visual content is complex and varied. At the same time, images and questions as two modalities have huge differences in semantics and information content, which makes visual question answering face huge challenges. Specifically, there are two main difficulties:

[0007] (1) How to effectively construct image visual features and question language features with strong representation capabilities to obtain semantically rich information: Traditional target detection models extract visual features based on objects (regions), which have the disadvantages of independent training of target detectors, long training time, complex visual concepts, and lack of global image information. Traditional grid features have the disadvantages of weak representation capabilities and lack of language-related semantic information. Therefore, constructing visual / language features that are suitable for end-to-end deep learning models and have strong representation capabilities is one of the bottlenecks of visual question answering performance.

[0008] (2) How to integrate, interact, and reason across modal information to improve the accuracy of answer output: Previous visual question answering methods usually use a bilinear pooling-based fusion method for visual features and language features, while the use of self-attention and cross-attention mechanisms can have more effective semantic integration and interaction capabilities for modal information and inter-modal information. The core difficulty in visual question answering algorithms is to automatically learn the connection between visual features and language features through deep stacked networks, fully integrate and interact within and between modalities, form more accurate cross-modal expressions, and then reason and output answers. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides a visual question answering method based on cross-modal pre-training feature enhancement. The present invention mainly includes two points:

[0010] 1. The semantically aligned local / global visual features and language features are constructed through the cross-modal pre-training model, and the relative spatial relationship features under the grid features are modeled as an efficient semantic expression of image,questions.

[0011] 2. A feature enhancement module and a deeply stacked language-side SA module and a visual-side RGA module are proposed to obtain richer composite semantic information. Finally, the comprehensive features are output through the local / global feature fusion module for accurate answer prediction.

[0012] A visual question answering method based on cross-modal pre-training feature enhancement includes the following steps:

[0013] Step (1): partitioning of data set;

[0014] First, the image set and question answer annotation set of visual question answering are divided into training set, validation set and test set;

[0015] Step (2): Construct local and global language features of the problem;

[0016] For an input question, first segment it according to the word segmentation table and fill it to a fixed number of segmentations T, then use the pre-trained word vector model to convert the word segmentation representation of the question into a word embedding containing semantic information Then, the local language features are obtained through the cross-modal pre-trained CLIP language encoder. T is the number of language feature words, d t is the dimension of the language feature of a single word; from the local language feature Q loc Select the question participle to indicate the end symbol <eos>The word feature corresponding to position l∈[1,T] Obtain the global language features of the problem through a linear transformation

[0017] Step (3): construct local and global visual features of the image;

[0018] For an input image, it is first preprocessed by using bicubic interpolation to scale the image to a fixed square size P×P, and then the image pixels are normalized; the feature map of the last layer is extracted using the cross-modal pre-trained CLIP visual encoder. The feature map is transformed into a grid feature form as the local visual feature of the image. G is the number of grids d v is the dimension of a single grid feature. Local visual feature V loc Further global visual features are obtained through attention pooling mechanism and linear transformation

[0019] Step (4): construct relative spatial relationship features;

[0020] Calculate the spatial coordinates of all grids based on the local visual features (grid features) of an input image G is the number of grids; the relative spatial relationship characteristics are obtained by calculating all grids pairwise.

[0021] Step (5): construct a deep neural network;

[0022] The deep neural network includes a language feature enhancement module, a visual feature enhancement module, a language side deep stacking SA module, a visual side deep stacking RGA module, a language feature attention ablation module, a visual feature attention ablation module, a local / global feature fusion module and an answer probability distribution prediction module;

[0023] Local language feature Q loc The fusion local language features are obtained by enhancing the fusion module through the language features Local visual features V loc The fused local visual features are obtained by visual feature enhancement fusion module Fusion of local language features Q fuse Get composite language features by deeply stacking SA modules on the language side Fusion of local visual features V fuse The composite visual features are obtained by deeply stacking RGA modules on the visual side. Composite language feature Q' fuse After the language feature attention ablation module, the comprehensive local language features are obtained Composite visual feature V' fuse After the visual feature attention ablation module, the comprehensive local visual features are obtained Comprehensive local language / visual featuresq loc 、v loc and global language / visual features q glo 、v glo Further output comprehensive features through local / global feature fusion module The answer linear projection matrix through the answer probability distribution prediction module The final output is the predicted answer probability A is the size of the candidate answer set;

[0024] Step (6): Construct loss function;

[0025] Input the predicted answer probability distribution output in step (5) together with the corresponding correct answer score distribution into the corresponding loss function to calculate the loss function value;

[0026] Step (7): training the model;

[0027] The deep neural network model is trained on the training set in step (1), and the model performance is verified on the validation set in each round; the back propagation algorithm is used to perform gradient back propagation on the deep neural network model parameters according to the loss function value generated by the loss function in step (6), and the optimization is continuously performed until the entire network model converges;

[0028] Step (8): Calculate the predicted answer value;

[0029] During the model validation or testing phase, the probability distribution of the predicted answers based on the deep neural network Select the word corresponding to the index with the largest probability value as the predicted answer.

[0030] Furthermore, the data set described in step (1) is divided as follows:

[0031] The dataset used for the visual question answering task is VQA-v2, and its images are all from the MS-COCO dataset; the dataset is divided into three subsets: train, val, and test, and the data volume of the three subsets accounts for approximately 40%, 20%, and 40% respectively.

[0032] Furthermore, the local and global language features of the constructed problem described in step (2) are as follows:

[0033] 2-1. Problem preprocessing;

[0034] For an input question, firstly segment the question text according to the BPE word segmentation table, and concatenate the text start symbol at the head <sot>, add the text terminator at the end <eot>Then fill the end of the terminator with 0 to a fixed number of word segments T to get the word segmentation representation of the question Each participle The value of is the word index in the BPE word table;

[0035] 2-2. Question word embedding;

[0036] Use the pre-trained word vector model to convert each word segment into a word embedding containing semantic information d t is the word embedding dimension of a single word, and all word segments are concatenated to obtain the word embedding representation of the question

[0037] 2-3. Extraction of local language features of questions;

[0038] The word embedding representation of the question is encoded by the language encoder in the cross-modal pre-trained CLIP model to obtain local language features. T is the number of language feature words, d t is the dimension of the language feature of a single word. The specific formula is as follows;

[0039] Q loc =CLIP TE (Q emb ) (Formula 1)

[0040] 2-4. Extraction of global language features of questions;

[0041] In the local language feature Q loc Select the question participle to indicate the end symbol <eos>The word feature corresponding to position l∈[1,T] Through a linear projection matrix Linear transformation obtains the global language characteristics of the problem The specific formula is as follows:

[0042]

[0043] Furthermore, the local and global visual features of the constructed image in step (3) are specifically as follows:

[0044] 3-1. Image preprocessing;

[0045] For an input image, first preprocess it and use bicubic interpolation to scale the image to a fixed square size P×P to obtain the scaled image Then the image pixels are normalized according to the three channels of RGB to obtain the normalized image. The specific formula for image normalization is as follows:

[0046]

[0047] where μ R , μ G , μ B are the fixed RGB channel means, σ R , σ G , σ B are fixed RGB channel standard deviations, They are the RGB pixel values ​​at the (x, y) coordinates of the image after scaling to a fixed size. are the RGB pixel values ​​at the normalized image (x,y) coordinates, x,y∈[0,P);

[0048] 3-2. Local visual feature extraction;

[0049] The preprocessed image I' is extracted through the visual encoder in the cross-modal pre-trained CLIP model to obtain the feature map of the last layer. The feature map is transformed into a grid feature as the local visual feature of the image. G is the number of grids G = g × g, d v is the dimension of a single grid feature, and the specific formula is as follows:

[0050] V loc =CLIP VE (I') (Formula 4)

[0051] 3-3. Global visual feature extraction;

[0052] First, the local visual feature V loc Take the average value in the grid number dimension to get the average visual feature The specific formula is as follows:

[0053]

[0054] The average visual features Spliced ​​to the local visual feature V loc head, plus a learnable position embedding matrix Get new local visual features The specific formula is as follows:

[0055]

[0056] Further, the local visual features with global information are obtained through the multi-head attention mechanism MHSA and linear transformation. The specific formula is as follows:

[0057]

[0058] in is the linear projection matrix;

[0059] Select the head of local visual features with global information As a global visual feature

[0060] Furthermore, the relative spatial relationship feature constructed in step (4) is specifically as follows:

[0061] 4-1. Grid space coordinate calculation;

[0062] For an image scaled to a fixed size P×P, it has G grid features, Each grid is 32 pixels wide and high; the spatial coordinates of all grids can be pre-calculated The specific formula is as follows:

[0063] B i =[x i,1 ,y i,1 ,x i,2 ,y i,2 ] (Formula 8)

[0064]

[0065] Where (x i,1 ,y i,1 ) is the coordinate of the upper left corner of grid i, (x i,2 ,y i,2 ) is the coordinate of the lower right corner of the grid i, i∈[0,G);

[0066] 4-2. Relative spatial relationship calculation;

[0067] Convert the grid space coordinates to the grid center point horizontal coordinate x c , the vertical coordinate y of the center point c , width w, height h format G is the number of grids, and the specific formula is as follows:

[0068]

[0069] The relative spatial relationship between the i-th grid and the j-th grid is characterized by The specific formula is as follows:

[0070] R i,j =[Δx i,j ,Δy i,j ,Δw i,j ,Δh i,j ]

[0071]

[0072] The total relative spatial relationship features are calculated for all grids in the image.

[0073] Furthermore, the construction of the deep neural network described in step (5) is as follows:

[0074] The deep neural network includes a language feature enhancement fusion module, a visual feature enhancement fusion module, a language side deep stacking SA module, a visual side deep stacking RGA module, a language feature attention ablation module, a visual feature attention ablation module, a local / global feature fusion module and an answer probability distribution prediction module.

[0075] 5-1. Language feature enhancement and fusion module;

[0076] The local language feature Q obtained in step 2-3 is loc Enhanced language features are obtained through language enhancement MLP The specific formula is as follows:

[0077]

[0078] in is the parameter matrix in the 2-layer MLP;

[0079] Enhanced language features Q enh and the original local language feature Q loc Further, the joint language feature is obtained by weighted summation with weight ratio α∈[0,1] The specific formula is as follows:

[0080] Q apt =αQ enh +(1-α)Q loc (Formula 12)

[0081] Finally, the combined language feature Q apt After the linear projection matrix Transformation to obtain fusion local language features The specific formula is as follows:

[0082]

[0083] 5-2. Visual feature enhancement fusion module;

[0084] The local visual feature V obtained in step 3-2 loc Enhanced visual features are obtained through visual enhancement MLP The specific formula is as follows:

[0085]

[0086] in is the parameter matrix in the 2-layer MLP;

[0087] Enhanced visual features V enh and local visual features V loc Further, the joint visual feature is obtained by weighted summation with weight ratio β∈[0,1] The specific formula is as follows:

[0088] V apt =βV enh +(1-β)V loc (Formula 15)

[0089] The final joint visual feature V apt After the linear projection matrix Transform to obtain fused local visual features The specific formula is as follows:

[0090]

[0091] 5-3. Deep stacking of SA modules on the language side;

[0092] Define the SA module. The specific formulas in the module are as follows:

[0093] Among them, MHSA is a multi-head self-attention unit, FFN is a 2-layer feedforward network, and LN is a layer normalization operation;

[0094] The MHSA formula is as follows:

[0095]

[0096] Among them, Q, K, and V are obtained by mapping X through the fully connected layer respectively;

[0097] The formula for FFN is as follows:

[0098] FFN(X')=FC d (Dropout(ReLU(FC 4d (X')))) (Formula 19)

[0099] Among them FC 4d Contains the linear projection matrix FC d Contains the linear projection matrix

[0100] The calculation formula for the nth SA module on the language side is as follows:

[0101] Z (n) =SA(Z (n-1) ) (Formula 20)

[0102] Where Z (0) =Q fuse , that is, integrating local language features Q fuse As the input of the first SA module, by stacking L layers of SA modules, the output Z of the last layer can be obtained after iterative reciprocating calculations. (L) As a composite language feature

[0103] 5-4. Deep stacking of RGA modules on the visual side;

[0104] First, convert the relative spatial relationship feature R into a relative spatial relationship embedding d r is the relative spatial relationship embedding dimension, and the specific formula is as follows:

[0105] R'=ReLU(RW R ) (Formula 21)

[0106] in Embed the linear projection matrix for relative spatial relations, and ReLU is the activation function;

[0107] Define the RGA module, the specific formula is as follows:

[0108]

[0109] Among them, MHRSA is a multi-head relational self-attention unit, MHGA is a multi-head guided attention unit, FFN is a 2-layer feedforward network, and LN is a layer normalization operation;

[0110] The MHRSA formula is as follows:

[0111]

[0112] Q, K, and V are obtained by mapping X through the fully connected layer respectively; It is obtained by R through full connection layer mapping, ReLU activation function, and log calculation;

[0113] The formula for MHGA is as follows:

[0114]

[0115] Among them, Q is obtained by mapping Y through the fully connected layer, and K and V are obtained by mapping X through the fully connected layer;

[0116] The formula for FFN is as follows:

[0117] FFN(X')=FC d (Dropout(ReLU(FC 4d (X')))) (Formula 25)

[0118] Among them FC 4d Contains the linear projection matrix FC d Contains the linear projection matrix

[0119] The calculation formula of the nth RGA module on the visual side is as follows:

[0120] Z (n) =RGA(Z (n-1) ,X,R') (Formula 26)

[0121] Where Z (0) =V fuse ,X=Q' fuse , that is, fusion of local visual features V fuse , compound language feature Q' fuse , the relative spatial relationship embedding R' is used as input, and by stacking L layers of RGA modules, the output Z of the last layer can be obtained after iterative reciprocating calculations (L) As a composite visual feature

[0122] 5-5. Language feature attention ablation module;

[0123] Composite language feature Q' fuse The attention weights of all word features are first calculated through the 2-layer MLP in the language feature attention ablation module The specific formula is as follows:

[0124]

[0125] in, They are the parameter matrices in the 2-layer MLP respectively;

[0126] Language attention weight w Q With the composite language feature Q' fuse Perform weighted summation to obtain comprehensive local language features The specific formula is as follows:

[0127]

[0128] 5-6. Visual feature attention ablation module;

[0129] Composite visual feature V' fuse The attention weights corresponding to all grid features are first calculated through the 2-layer MLP in the visual feature attention ablation module The specific formula is as follows:

[0130]

[0131] in, are the parameter matrices in the 2-layer MLP respectively;

[0132] Visual attention weight w V With composite visual feature V' fuse Perform weighted summation to obtain comprehensive local visual features The specific formula is as follows:

[0133]

[0134] 5-7. Local / global feature fusion module;

[0135] Comprehensive local visual features v loc , global visual features v loc , comprehensive local language features q loc and the global language feature q glo , after the local / global feature fusion module, the comprehensive features are obtained The details are as follows:

[0136]

[0137] Where LN is the layer normalization operation, is the linear projection matrix of local visual features, is the global visual feature linear projection matrix, is the linear projection matrix of local language features, is the linear projection matrix of global language features;

[0138] 5-8.Answer probability distribution prediction module;

[0139] Comprehensive feature f fuse By answer linear projection matrix Linear transformation outputs the predicted answer probability distribution The details are as follows:

[0140]

[0141] Where σ is the sigmoid function, which transforms the predicted answer probability into the range of (0,1).

[0142] Furthermore, the loss function described in step (6) is as follows:

[0143] Calculate the predicted answer probability distribution of the output The corresponding true answer score distribution The gap between them, here we use the binary cross entropy (BCE, binary cross entropy) loss function. The specific formula is as follows:

[0144]

[0145] Furthermore, the training model described in step (7) is as follows:

[0146] The loss value generated by the loss function in step (6) The back propagation algorithm is used to perform gradient backpropagation on the model parameters of the deep neural network in step (5) and continuously optimize until the entire network model converges.

[0147] Furthermore, the answer prediction value calculation in step (8) is as follows:

[0148] Probability distribution of predicted answers based on the output of the deep neural network Select the index with the largest probability value The corresponding word is used as the predicted answer. The specific formula is as follows:

[0149]

[0150] The beneficial effects of the present invention are as follows:

[0151] This paper proposes a visual question answering method based on cross-modal pre-trained feature enhancement (XPFE-VQA), which models relative spatial relationship features, uses semantically aligned local / global visual features and language features, combines feature enhancement modules and deeply stacked language-side SA modules and visual-side RGA modules to obtain richer composite semantic information, and finally outputs comprehensive features through local / global feature fusion modules for answer prediction. This method has achieved significant improvement in visual question answering tasks, surpassing many methods that use traditional object features or grid features as visual features, and even outperforming many models that use the visual-language "pre-training-fine-tuning" paradigm. BRIEF DESCRIPTION OF THE DRAWINGS

[0152] Figure 1 Schematic diagram of the overall network architecture (XPFE-VQA) of an embodiment of the present invention;

[0153] Figure 2 Schematic diagram of the feature enhancement module structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0154] The detailed parameters of the present invention are further described in detail below.

[0155] like Figure 1 , 2 As shown, the present invention provides a visual question answering method based on cross-modal pre-training feature enhancement.

[0156] Step (1) The data set is divided as follows:

[0157] The dataset used for the visual question answering task is VQA-v2, whose images are all from the MS-COCO dataset; the dataset is divided into three subsets: train, val, and test. The train set contains 443,757 questions about 82,783 images, the val set contains 214,354 questions about 40,504 images, and the test set contains 447,793 questions about 81,434 images.

[0158] Step (2) constructs the local and global language features of the problem, as follows:

[0159] 2-1. The question text is segmented to obtain the segmentation representation of the question

[0160] 2-2. Obtain word embedding representation of the question through the word vector model

[0161] 2-3. Use 12-layer CLIP-Transformer language encoder to encode local language features

[0162] 2-4. Through the linear projection matrix Linear transformation obtains the global language characteristics of the problem

[0163] Step (3) constructs the local and global visual features of the image, as follows:

[0164] 3-1. The input image is scaled to a fixed square size of 512×512 using bicubic interpolation. The fixed RGB channel means in the image normalization operation are μ R =0.48145466, μ G =0.4578275, μ B =0.40821073, the fixed RGB channel standard deviations are σ R =0.26862954,σ G =0.26130258,σ B =0.27577711;

[0165] 3-2. Use CLIP-RN50×16 visual encoder to encode local visual features The corresponding grid number G = 16 × 16 = 256;

[0166] 3-3. Average visual features Learnable position embedding matrix New local visual features Local visual features with global information Through the linear projection matrix Linear transformation to obtain global visual features

[0167] Step (4) constructs relative spatial relationship features, as follows:

[0168] 4-1. Pre-calculate the spatial coordinates of all grids

[0169] 4-2. Transformed spatial coordinates The total relative spatial relationship characteristics are obtained by calculating the grids pairwise.

[0170] Step (5) constructs a deep neural network, as follows:

[0171] The deep neural network includes a language feature enhancement fusion module, a visual feature enhancement fusion module, a language side deep stacking SA module, a visual side deep stacking RGA module, a language feature attention ablation module, a visual feature attention ablation module, a local / global feature fusion module and an answer probability distribution prediction module.

[0172] 5-1. The parameter matrices in the 2-layer language enhancement MLP in the language feature enhancement fusion module are The local language feature Q loc Enhanced language features are obtained through language enhancement MLP By weight ratio Weighted summation to obtain joint language features After the linear projection matrix Transformation to obtain fusion local language features

[0173] 5-2. The parameter matrices of the 2-layer visual enhancement MLP in the visual feature enhancement fusion module are The local visual features V loc Enhanced visual features are obtained through visual enhancement MLP By weight ratio Weighted summation to obtain joint visual features After the linear projection matrix Transform to obtain fused local visual features

[0174] 5-3. Fusion of local language features Q fuse The composite language features are obtained by deeply stacking 6 layers of SA modules on the language side. In the SA module, MHSA uses 8 heads, each with a dimension of 96; in FFN, FC 4d Linear projection matrix FC d Linear projection matrix The dimension of LN is 768;

[0175] 5-4. Embedding the Linear Projection Matrix through Relative Spatial Relationship Get relative spatial relationship embedding Fusion of local visual features V fuse The composite visual features are obtained by deeply stacking 6 layers of RGA modules on the visual side. In the RGA module, MHRSA and MHGA both use 8 heads, and the dimension of each head is 96; in FFN, FC 4d Linear projection matrix FC d Linear projection matrix The dimension of LN is 768;

[0176] 5-5. Parameter matrix in the 2-layer MLP used by the language feature attention ablation module Get the attention weights of all word features The comprehensive local language features are obtained by weighted summation

[0177] 5-6. Parameter matrix in the 2-layer MLP used in the visual feature attention ablation module Get the attention weights corresponding to all grid features The comprehensive local visual features are obtained by weighted summation

[0178] 5-7. Local visual feature linear projection matrix Global visual feature linear projection matrix Local language feature linear projection matrix Global language feature linear projection matrix The dimension of LN is 1536; the comprehensive local visual features v loc , global visual features v loc , comprehensive local language features q loc and the global language feature q glo , after the local / global feature fusion module, the comprehensive features are obtained

[0179] 5-8. Answer linear projection matrix through answer probability distribution prediction module Output predicted answer probability distribution The size of the candidate answer set is 3129.

[0180] Step (6) calculates the loss function as follows:

[0181] Calculate the predicted answer probability distribution of the output The corresponding true answer score distribution The gap between the true answer and the multi-label score y a ∈{0,0.3,0.6,0.9,1}. The loss function uses the binary cross entropy (BCE) loss function.

[0182] Step (7) training the model, as follows:

[0183] The model is trained on the training set in step (1), and the model performance is verified on the validation set in each round; the loss value generated by the loss function in step (6) The back propagation algorithm is used to perform gradient backpropagation on the model parameters of the deep neural network in step (5) and continuously optimize until the entire network model converges.

[0184] Step (8) Calculation of the predicted answer value is as follows:

[0185] During the model validation or testing phase, the predicted answer probability distribution output from step (5) is Select the index with the largest probability value The corresponding word is used as the predicted answer.< / eos> < / eot> < / sot> < / eos>

Claims

1. A visual question answering method based on cross-modal pre-training feature enhancement, characterized in that: The steps include: Step (1): partitioning of data set; First, the image set and question answer annotation set of visual question answering are divided into training set, validation set and test set; Step (2): Construct local and global language features of the problem; For an input question, first segment it according to the word segmentation table and fill it to a fixed number of segmentations T, then use the pre-trained word vector model to convert the word segmentation representation of the question into a word embedding containing semantic information Then, the local language features are obtained through the cross-modal pre-trained CLIP language encoder. T is the number of language feature words, d t is the dimension of the language feature of a single word; from the local language feature Q loc Select the question participle to indicate the end symbol <eos>The word feature corresponding to position l∈[1,T] Obtain the global language features of the problem through a linear transformation < / eos> Step (3): construct local and global visual features of the image; For an input image, it is first preprocessed by using bicubic interpolation to scale the image to a fixed square size P×P, and then the image pixels are normalized; the feature map of the last layer is extracted using the cross-modal pre-trained CLIP visual encoder. The feature map is transformed into a grid feature form as the local visual feature of the image. G is the number of grids d v is the dimension of a single grid feature; local visual feature V loc Further global visual features are obtained through attention pooling mechanism and linear transformation Step (4): construct relative spatial relationship features; Calculate the spatial coordinates of all grids based on the local visual features of an input image G is the number of grids; the relative spatial relationship characteristics are obtained by calculating all grids pairwise. Step (5): construct a deep neural network; The deep neural network includes a language feature enhancement module, a visual feature enhancement module, a language side deep stacking SA module, a visual side deep stacking RGA module, a language feature attention ablation module, a visual feature attention ablation module, a local / global feature fusion module and an answer probability distribution prediction module; Local language feature Q loc The fusion local language features are obtained by enhancing the fusion module through the language features Local visual features V loc The fused local visual features are obtained by visual feature enhancement fusion module Fusion of local language features Q fuse Get composite language features by deeply stacking SA modules on the language side Fusion of local visual features V fuse The composite visual features are obtained by deeply stacking RGA modules on the visual side. Composite language feature Q' fuse After the language feature attention ablation module, the comprehensive local language features are obtained Composite visual feature V' fuse After the visual feature attention ablation module, the comprehensive local visual features are obtained Comprehensive local language / visual featuresq loc 、v loc and global language / visual features q glo 、v glo Further output comprehensive features through local / global feature fusion module The answer linear projection matrix through the answer probability distribution prediction module The final output is the predicted answer probability A is the size of the candidate answer set; Step (6): Construct loss function; Input the predicted answer probability distribution output in step (5) together with the corresponding correct answer score distribution into the corresponding loss function to calculate the loss function value; Step (7): training the model; The deep neural network model is trained on the training set in step (1), and the model performance is verified on the validation set in each round; the back propagation algorithm is used to perform gradient back propagation on the deep neural network model parameters according to the loss function value generated by the loss function in step (6), and the optimization is continuously performed until the entire network model converges; Step (8): Calculate the predicted answer value; During the model validation or testing phase, the probability distribution of the predicted answers based on the deep neural network Select the word corresponding to the index with the largest probability value as the predicted answer.

2. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 1, characterized in that: The data set described in step (1) is divided as follows: The dataset used for the visual question answering task is VQA-v2, and its images are all from the MS-COCO dataset; the dataset is divided into three subsets: train, val, and test, and the data volume of the three subsets accounts for approximately 40%, 20%, and 40% respectively.

3. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 2, characterized in that: The local and global language features of the construction problem described in step (2) are as follows: 2-1. Problem preprocessing; For an input question, firstly segment the question text according to the BPE word segmentation table, and concatenate the text start symbol at the head <sot>, add the text terminator at the end <eot>Then fill the end of the terminator with 0 to a fixed number of word segments T to get the word segmentation representation of the question Each participle The value of is the word index in the BPE word table;< / eot> < / sot> 2-2. Question word embedding; Use the pre-trained word vector model to convert each word segment into a word embedding containing semantic information d t is the word embedding dimension of a single word, and all word segments are concatenated to obtain the word embedding representation of the question 2-3. Extraction of local language features of questions; The word embedding representation of the question is encoded by the language encoder in the cross-modal pre-trained CLIP model to obtain local language features. T is the number of language feature words, d t is the dimension of the language feature of a single word. The specific formula is as follows; Q loc =CLIP TE (Q emb ) (Formula 1) 2-4. Extraction of global language features of questions; In the local language feature Q loc Select the question participle to indicate the end symbol <eos>The word feature corresponding to position l∈[1,T] Through a linear projection matrix Linear transformation obtains the global language characteristics of the problem The specific formula is as follows:< / eos> 4. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 3, characterized in that: The local and global visual features of the constructed image in step (3) are as follows: 3-1. Image preprocessing; For an input image, first preprocess it and use bicubic interpolation to scale the image to a fixed square size P×P to obtain the scaled image Then the image pixels are normalized according to the three channels of RGB to obtain the normalized image. The specific formula for image normalization is as follows: where μ R , μ G , μ B are the fixed RGB channel means, σ R , σ G , σ B are fixed RGB channel standard deviations, They are the RGB pixel values ​​at the (x, y) coordinates of the image after scaling to a fixed size. are the RGB pixel values ​​at the normalized image (x,y) coordinates, x,y∈[0,P); 3-2. Local visual feature extraction; The preprocessed image I' is extracted through the visual encoder in the cross-modal pre-trained CLIP model to obtain the feature map of the last layer. The feature map is transformed into a grid feature as the local visual feature of the image. G is the number of grids G = g × g, d v is the dimension of a single grid feature, and the specific formula is as follows: V loc =CLIP VE (I') (Formula 4) 3-3. Global visual feature extraction; First, the local visual feature V loc Take the average value in the grid number dimension to get the average visual feature The specific formula is as follows: The average visual features Spliced ​​to the local visual feature V loc head, plus a learnable position embedding matrix Get new local visual features The specific formula is as follows: Further, the local visual features with global information are obtained through the multi-head attention mechanism MHSA and linear transformation. The specific formula is as follows: in is the linear projection matrix; Select the head of local visual features with global information As a global visual feature 5. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 4, characterized in that: The relative spatial relationship feature constructed in step (4) is specifically as follows: 4-1. Grid space coordinate calculation; For an image scaled to a fixed size P×P, it has G grid features, G=g×g, Each grid is 32 pixels wide and high; Pre-calculate the spatial coordinates of all grids The specific formula is as follows: B i =[x i,1 ,y i,1 ,x i,2 ,y i,2 ] Where (x i,1 ,y i,1 ) is the coordinate of the upper left corner of grid i, (x i,2 ,y i,2 ) is the coordinate of the lower right corner of the grid i, i∈[0,G); 4-2. Relative spatial relationship calculation; Convert the grid space coordinates to the horizontal coordinate x of the grid center point c , the vertical coordinate y of the center point c , width w, height h format G is the number of grids, and the specific formula is as follows: B′ i =[x i,c ,y i,c ,w i ,h i ] The relative spatial relationship between the i-th grid and the j-th grid is characterized by The specific formula is as follows: R i,j =[Δx i,j ,Δy i,j ,Δw i,j ,Δh i,j ] The total relative spatial relationship features are calculated for all grids in the image.

6. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 5, characterized in that: The construction of the deep neural network described in step (5) is as follows: The deep neural network includes a language feature enhancement fusion module, a visual feature enhancement fusion module, a language side deep stacking SA module, a visual side deep stacking RGA module, a language feature attention ablation module, a visual feature attention ablation module, a local / global feature fusion module and an answer probability distribution prediction module; 5-1. Language feature enhancement and fusion module; The local language feature Q obtained in step 2-3 is loc Enhanced language features are obtained through language enhancement MLP The specific formula is as follows: in is the parameter matrix in the 2-layer MLP; Enhanced language features Q enh and the original local language feature Q loc Further, the joint language feature is obtained by weighted summation with weight ratio α∈[0,1] The specific formula is as follows: Q apt =αQ enh +(1-α)Q loc (formula 12) Finally, the combined language feature Q apt After the linear projection matrix Transformation to obtain fusion local language features The specific formula is as follows: 5-2. Visual feature enhancement fusion module; The local visual feature V obtained in step 3-2 loc Enhanced visual features are obtained through visual enhancement MLP The specific formula is as follows: in is the parameter matrix in the 2-layer MLP; Enhanced visual features V enh and local visual features V loc Further weighted summation is performed by weight ratio β∈0,1] to obtain the joint visual feature The specific formula is as follows: V apt = βV enh + (1 - β)V loc (Equation 15) The final joint visual feature V apt After the linear projection matrix Transform to obtain fused local visual features The specific formula is as follows: 5-3. Deep stacking of SA modules on the language side; Define the SA module. The specific formulas in the module are as follows: Among them, MHSA is a multi-head self-attention unit, FFN is a 2-layer feedforward network, and LN is a layer normalization operation; The MHSA formula is as follows: Among them, Q, K, and V are obtained by mapping X through the fully connected layer respectively; The formula for FFN is as follows: Among them FC 4d Contains the linear projection matrix FC d Contains the linear projection matrix The calculation formula for the nth SA module on the language side is as follows: Z (n) = SA(Z (n-1) ) (Equation 20) Where Z (0) =Q fuse , that is, integrating local language features Q fuse As the input of the first SA module, by stacking L layers of SA modules, the output Z of the last layer is obtained after iterative reciprocating calculations. (L) As a composite language feature 5-4. Deep stacking of RGA modules on the visual side; First, convert the relative spatial relationship feature R into a relative spatial relationship embedding d r is the relative spatial relationship embedding dimension, and the specific formula is as follows: R'=ReLU(RW R ) (Formula 21) in Embed the linear projection matrix for relative spatial relations, and ReLU is the activation function; Define the RGA module, the specific formula is as follows: Among them, MHRSA is a multi-head relational self-attention unit, MHGA is a multi-head guided attention unit, FFN is a 2-layer feedforward network, and LN is a layer normalization operation; The MHRSA formula is as follows: Q, K, and V are obtained by mapping X through the fully connected layer respectively; It is obtained by R through full connection layer mapping, ReLU activation function, and log calculation; The formula for MHGA is as follows: Among them, Q is obtained by mapping Y through the fully connected layer, and K and V are obtained by mapping X through the fully connected layer; The formula for FFN is as follows: FFN(X')=FC d (Dropout(ReLU(FC 4d (X')))) (Formula 25) Among them FC 4d Contains the linear projection matrix FC d Contains the linear projection matrix The calculation formula of the nth RGA module on the visual side is as follows: Z (n) =RGA(Z (n-1) ,X,R') (Formula 26) Where Z (0) =V fuse ,X=Q' fuse , that is, fusion of local visual features V fuse , compound language feature Q' fuse , the relative spatial relationship embedding R' is used as input, and the output Z of the last layer is obtained by stacking L layers of RGA modules and iteratively calculating back and forth (L) As a composite visual feature 5-5. Language feature attention ablation module; Composite language feature Q' fuse The attention weights of all word features are first calculated through the 2-layer MLP in the language feature attention ablation module The specific formula is as follows: in, They are the parameter matrices in the 2-layer MLP respectively; Language attention weight w Q With the composite language feature Q' fuse Perform weighted summation to obtain comprehensive local language features The specific formula is as follows: 5-6. Visual feature attention ablation module; Composite visual feature v' fuse The attention weights corresponding to all grid features are first calculated through the 2-layer MLP in the visual feature attention ablation module The specific formula is as follows: in, They are the parameter matrices in the 2-layer MLP respectively; Visual attention weight w V With composite visual feature V' fuse Perform weighted summation to obtain comprehensive local visual features The specific formula is as follows: 5-7. Local / global feature fusion module; Comprehensive local visual featuresv loc , global visual features v loc , comprehensive local language features q loc and the global language feature q glo , after the local / global feature fusion module, the comprehensive features are obtained The details are as follows: Where LN is the layer normalization operation, is the linear projection matrix of local visual features, is the global visual feature linear projection matrix, is the linear projection matrix of local language features, is the linear projection matrix of global language features; 5-8.Answer probability distribution prediction module; Comprehensive feature f fuse By answer linear projection matrix Linear transformation outputs the predicted answer probability distribution The details are as follows: Where σ is the sigmoid function, which transforms the predicted answer probability into the range of (0,1).

7. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 6, characterized in that: The loss function described in step (6) is as follows: Calculate the predicted answer probability distribution of the output The corresponding true answer score distribution The gap between them, here we use the binary cross entropy loss function; the specific formula is as follows:

8. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 7, characterized in that: The training model described in step (7) is as follows: The loss value generated by the loss function in step (6) The back propagation algorithm is used to perform gradient backpropagation on the model parameters of the deep neural network in step (5) and continuously optimize until the entire network model converges.

9. The visual question answering method based on cross-modal pre-training feature enhancement according to claim 8, characterized in that: The answer prediction value calculation described in step (8) is as follows: Probability distribution of predicted answers based on the output of the deep neural network Select the index with the largest probability value The corresponding word is used as the predicted answer. The specific formula is as follows:

Images