Lightweight visual question answering method based on bidirectional separable deep self-attention network

Abstract

The present invention discloses a lightweight visual question-answering method based on a bidirectionally separable deep self-attention network, and proposes a bidirectionally separable deep self-attention network. Based on the designed bidirectional strategy of separability in both width and depth, and using the "deep and narrow" filtering principle to further select reasonable sub-models, combined with the proposed self-distillation algorithm, each sub-model in the network has the application capability of visual question-answering tasks. This bidirectionally separable deep self-attention model can dynamically select a suitable sub-model to predict the answer according to the current computing resources, achieve a balance between accuracy and latency, and ensure the accuracy of the predicted answer while giving users a good experience.

Description

Technical Field

[0001] The present invention belongs to the field of visual question answering, and specifically relates to a lightweight visual question answering method (Visual Question Answering) based on a bidirectionally separable deep self-attention network. Background Art

[0002] The visual question answering task is a hot research issue in multimodal learning tasks. It is a learning task involving computer vision and natural language processing. Its goal is to output the corresponding predicted answer for a given image and a free-form question related to the image after being processed by the visual question answering model. Unlike single-modal tasks, the visual question answering task as a multimodal task requires not only the correct understanding of information from different modalities, but also the understanding of the associated information between modalities, which is usually more complex and difficult. This task has a wide range of applications in real life. For example, it can help people with visual impairments to obtain online image information more conveniently; at the same time, it can promote the better development of human-computer interaction systems and enhance user experience; it can also improve the machine's understanding of images and enhance image retrieval capabilities.

[0003] The deep self-attention network was originally proposed for machine translation tasks in the field of natural language processing. Its core architecture is composed of multiple stacked self-attention layers, each of which can construct complex and dense interactions between input features. The deep self-attention network achieved the best results in machine translation tasks and quickly attracted the attention of researchers in the field of artificial intelligence, and was applied to various sub-fields of artificial intelligence, including the field of visual question answering. Because the deep self-attention network can better learn the interactive information between visual and text features, this network architecture has become the mainstream network structure in the field of visual question answering. However, while the deep self-attention network brings performance improvements, due to its computational complexity, it also puts forward new requirements for computing resources and storage space, which will bring a serious problem: deploying these models on mobile devices requires relying on GPU cloud servers, and it is difficult to directly use the limited computing resources of the mobile terminal, which will cause a waste of resources and is not environmentally friendly. At present, there is no lightweight model in the field of visual question answering, which brings challenges to the deployment of models and also hinders users from enjoying the convenience brought by artificial intelligence applications.

[0004] In order to cope with the new challenge of difficult deployment of deep learning models, some model compression methods have emerged. In the field of single-modal tasks such as computer vision and natural language processing, some compression models based on weight sharing, knowledge distillation, pruning, quantization and other methods have gradually emerged, which compress the model to a certain ratio to achieve a balance between computational complexity and accuracy. However, these methods are usually compressed to a fixed ratio and can only obtain a lightweight model of a fixed size. Today, there are many types of mobile devices, and the computing performance of different mobile devices varies greatly. Even the same mobile device can provide different computing resources under different load conditions and different power levels. If a lightweight model is designed for a mobile device or a load condition, the model training cost will be proportional to the number of models, and a mobile device needs to save multiple models to cope with various scenarios, and its storage cost is also very large.

[0005] Recently, a slimmable and splittable neural network has provided a new idea: use only one model to deal with multiple scenarios. When computing resources are sufficient, take most of the model for forward propagation and prediction to obtain higher accuracy; when computing resources are limited, take a small part of the model parameters for prediction, sacrificing a little accuracy for inference speed. If this idea can be used and an efficient and reasonable splitting and training strategy can be designed for the mainstream model structure deep self-attention network in the field of visual question answering, it will make new contributions to the implementation of visual question answering models.

[0006] In summary, how to design an efficient and separable deep self-attention network and apply it to the field of visual question answering is a topic worthy of in-depth research. This patent intends to explore several key points in this task, solve the difficulties and key points of the current methods, and form a complete, efficient and lightweight visual question answering method. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a lightweight visual question answering method based on a bidirectionally separable deep self-attention network. The present invention mainly includes two points:

[0008] 1. By analyzing the internal structure of the deep self-attention network, an efficient and reasonable width and depth segmentation strategy is designed, and the two strategies are combined to form a bidirectional strategy that can be segmented in both width and depth. For the sub-models segmented by the bidirectional strategy, this paper proposes a "deep and narrow" filtering principle and further selects some better and efficient sub-structures.

[0009] 2. The above-mentioned bidirectional segmentation strategy and filtering principle are combined with the existing visual question answering model based on deep self-attention, and an efficient self-distillation training strategy is proposed, so that each sub-model can be fully trained, and finally a bidirectional separable deep self-attention visual question answering model is obtained.

[0010] The present invention proposes a lightweight visual question answering method (Visual Question Answering) based on a bidirectionally separable deep self-attention network. The core method is to analyze the internal structure of the deep self-attention network, propose an efficient and reasonable width and depth segmentation strategy, and combine these two single-dimensional segmentation strategies to form a bidirectional strategy that can be segmented in width and depth. At the same time, for the substructures segmented by this bidirectional strategy, the present invention proposes a "deep and narrow" filtering principle, and further selects some superior and efficient substructures. This filtering principle can not only improve the performance of each substructure, but also does not require additional screening when the model is deployed and can be put into use directly, avoiding the post-processing process, which is simple and easy to use. In addition, an efficient self-distillation training strategy is also proposed, so that each sub-model can be fully trained. This method can be combined with any existing visual question-answering model based on a deep self-attention network. Through training, a bidirectionally separable deep self-attention network is formed. Each sub-model in the network has the application capability of visual question-answering tasks. When the model is deployed on some edge devices with limited resources and large performance fluctuations, this bidirectionally separable deep self-attention model can dynamically select the appropriate sub-model to predict the answer based on the current computing resources, achieving a balance between accuracy and latency, and ensuring the accuracy of the predicted answer while giving users a good experience.

[0011] A lightweight visual question answering method based on a bidirectional separable deep self-attention network includes the following steps:

[0012] Step (1): Divide the data set;

[0013] Step (2): construct visual features of the image;

[0014] For a given image, use the existing trained object detection network to detect the number m of candidate boxes in the image and their positions; for each candidate box, input the image corresponding to the candidate box area into the object detection network again, and extract the features before inputting into the network classification layer as the features of the candidate box. Then concatenate the features extracted from each candidate box to form the visual features of the given image. In order to make the image feature dimension match the deep self-attention network, a learnable linear transformation is finally used to further process the image features and map them to the D-dimensional space;

[0015] Step (3): Construct the semantic features of the question;

[0016] For a given problem, use the trained word vector model to extract semantic features for each word in the problem, and then splice the extracted single-word semantic features to form the problem semantic features. To make the dimension of the problem semantic features match that of the deep self-attention network, finally, a learnable linear transformation is used to further process the problem features and map them to the D-dimensional space;

[0017] Step (4): Construct the deep self-attention network;

[0018] The deep self-attention network is stacked by multiple self-attention layers. Each self-attention layer is divided into two parts: the multi-head attention module and the feed-forward layer. Use this deep self-attention network to construct the teacher network for guiding training and the final bidirectional splittable deep self-attention network. To make the input features match the dimensions of each sub-model in the bidirectional splittable deep self-attention network, this network accepts features of dimension D as input and maps the input features to the d dimension through a linear projection transformation.

[0019] Step (5): Design the width splitting strategy;

[0020] Each self-attention layer in the deep self-attention network is composed of multiple parameter matrices. To adapt to input features of different dimensions, each parameter matrix needs to be split to match inputs of different dimensions and output output features of appropriate dimensions. For input features of dimension d, in order to maintain the original structural ratio of the self-attention layer, the width splitting strategy is used to make the output feature dimension still d. It should be noted that for different sub-models with input features of different dimensions, the parameter matrices in the self-attention layer are shared. When d is smaller, the number of shared parameters is smaller; when d is equal to the original input dimension D, the parameter matrix is not split;

[0021] Step (6): Design the depth splitting strategy;

[0022] The deep self-attention network is stacked by multiple self-attention layers. Denote the number of layers as L. When the number of layers l of the sub-model < L, select l layers in this deep self-attention network according to the depth splitting strategy and attribute them to the sub-model itself. A simple and effective depth splitting strategy is proposed. Under different layer settings of the sub-model, it can select relatively important self-attention layers as much as possible, so as to improve the final accuracy of different sub-models;

[0023] Step (7): Combine the two splitting strategies and design the filtering principle;

[0024] Through the design of steps (5) and (6), each sub-model has a width d and a depth l. Under the same number of parameters and calculations, a deep and narrow sub-model will be more efficient and have a more reasonable structure than a shallow and wide sub-model. A "deep and narrow" filtering principle is proposed. Before model training, some sub-models with many layers and low width are selected, and those sub-models with few layers and high width are directly discarded. Through this filtering principle, the candidate set of sub-model structures is obtained.

[0025] Step (8): Design a self-distillation training algorithm and train the model;

[0026] For the candidate set of sub-model structures obtained in step (7), A self-distillation training strategy is proposed to enable each sub-model to be fully trained. First, a teacher network is trained using the deep self-attention network in step (4), and a bidirectional separable deep self-attention network is constructed. When training the sub-models in the bidirectional separable deep self-attention network, the image and question are first input into the teacher network to obtain their prediction vectors, called soft labels, and the candidate set is sampled during training through the sub-model sampling strategy. The sub-model in the training set uses this soft label as the supervision label of the sampled sub-model for training;

[0027] Step (9): Model deployment and application;

[0028] Furthermore, the data set described in step (1) is divided as follows:

[0029] The dataset described uses the VQA-v2 dataset, which is further divided into three subsets: training set, validation set, and test set. The training set is used to train the model, the validation set is used to locally verify the model convergence, and the test set is used for the final model performance evaluation.

[0030] Furthermore, the visual features of the constructed image in step (2) are specifically as follows:

[0031] For a given image, the number m of candidate boxes and their positions in the image are inferred using the existing trained Faster R-CNN object detection network, and the image area corresponding to each candidate box is input into the Faster R-CNN object detection network to extract its visual features. For the i-th candidate box, its corresponding visual feature is The visual features corresponding to the entire image It is composed of the visual features corresponding to each candidate box. The specific formula is as follows:

[0032] X image =[x1, x2, ..., x i, ..., x m ] (Formula 1)

[0033] Then, a learnable linear transformation is used For image feature X image After further processing, it is mapped to D-dimensional space to obtain the final image visual features. The specific formula is as follows:

[0034] X input =Linear(X image ) (Formula 2)

[0035] Furthermore, the semantic features of the constructed question described in step (3) are as follows:

[0036] For a given question, which contains n words, each word is input into the pre-trained GloVe word vector model to extract its semantic features. For the jth word, its corresponding semantic feature is The semantic features corresponding to the entire question It is composed of the semantic features corresponding to each word. The specific formula is as follows:

[0037] Y question =[y1,y2,...,y j , ..., y n ] (Formula 3)

[0038] Then, a learnable linear transformation is used The question semantic feature Y question After further processing, it is mapped to D-dimensional space to obtain the final semantic features of the question. The specific formula is as follows:

[0039] Y input =Linear(Y question ) (Formula 4)

[0040] Furthermore, the construction of the deep self-attention network described in step (4) is as follows:

[0041] The deep self-attention network is composed of multiple stacked self-attention layers, each of which is divided into two parts: a multi-head attention module and a feedforward layer. The deep self-attention network is used to construct a teacher network to guide the training and the final bidirectional separable deep self-attention network. The teacher network and the bidirectional separable deep self-attention network both use the same structure of the deep self-attention network. In order to make the input features match the dimensions of each sub-model in the bidirectional separable deep self-attention network, the deep self-attention network accepts image visual features of dimension D. and question semantic features As input, and through a linear projection transformation, the input features are mapped to d dimensions.

[0042] 4-1. Multi-head attention module;

[0043] For a given query feature Key Features And the value feature The multi-head attention module uses H parallel attention functions to calculate the features The specific formula is as follows:

[0044] F mha =MHA(Q,K,V)=[head1,head2,…,head H ]W 0 (Formula 5)

[0045]

[0046] in represents the mapping matrix of the h-th attention head, D H For the dimension of each attention head, we can use D H =D / H is calculated. In addition, It is used to further process the output features of the multi-head attention function. For the attention calculation method ATT, the specific formula is as follows:

[0047]

[0048] 4-2. Feedforward layer;

[0049] The feedforward layer consists of two layers of perceptrons, which perform nonlinear transformation on the output features of the multi-head attention module. Output Features The specific formula is as follows:

[0050]

[0051] in is the linear transformation projection matrix.

[0052] 4-3. Self-attention layer;

[0053] Each self-attention layer consists of the above-mentioned multi-head attention module and feed-forward layer. For a given input F input , output feature F output , the specific formula is as follows:

[0054]

[0055]

[0056] Where LN represents layer normalization.

[0057] 4-4. Stacking self-attention layers;

[0058] A deep self-attention network is composed of multiple stacked self-attention layers. The feature dimension does not change after the self-attention layer. Therefore, multiple self-attention layers can be connected in series to construct a deep self-attention network model. The specific formula is as follows:

[0059] Model = [Layer (1) , Layer (2) ,…,Layer (L) ] (Formula 11)

[0060] Where L is the number of self-attention layers.

[0061] Furthermore, the width segmentation strategy described in step (5) is as follows:

[0062] For the parameter matrix W in multi-head attention Q , W K , And the input features of dimension d, keep the size of each attention head D H The input matching dimension D and the number of attention heads H of the corresponding parameter matrix are changed. The final segmented parameter matrix W Q , W K , in Indicates the number of attention heads after segmentation. The parameter matrix W in other self-attention layers 0 , W1, W2 adopt the same strategy, so that the final segmented parameter matrix

[0063] Furthermore, the depth segmentation strategy described in step (6) is as follows:

[0064] For a deep self-attention network with L layers, the index of each layer is [1, 2, ..., L]. The present invention believes that the self-attention layers closer to the input and output are more important. This means that the middle layers are relatively less important. When the number of layers of the sub-model is l<L, it will start to discard the middle layers first. The specific operation is to sort each layer from the most important to the least important, and get the layer index as For a sub-model with l layers, take the first l items of the sorted layer index After reordering, restore to the original layer index order It is the depth segmentation strategy of the final l-layer sub-model.

[0065] Furthermore, the two segmentation strategies described in step (7) are combined to design filtering principles, which are as follows:

[0066] For a given width ratio candidate set and the depth ratio candidate set By combining the candidate sets of these two dimensions, we get the preliminary sub-model structure candidate set Each sub-model structure in In order to easily express the "deep and narrow" filtering principle, a two-dimensional index matrix is ​​defined To further process the preliminary sub-model candidate set I(d, l) = 1 means that the sub-model a(d, l) is selected, and I(d, l) = 0 means that the sub-model a(d, l) is discarded. The index matrix I is first initialized to all 1 values, and then the lower triangular part is converted to 0 values. Finally, the selected sub-model The specific definitions are as follows:

[0067]

[0068] Furthermore, the self-distillation training algorithm described in step (8) is specifically as follows:

[0069] Define the teacher network constructed by the deep self-attention network as M teacher , the bidirectional separable deep self-attention network is M DST , by training the teacher network M teacher Get its parameter weight θ and use this weight to initialize the bidirectional separable deep self-attention network M DST The weight θ DST . Through the sub-model sampling strategy, the candidate set is sampled during training The sub-model sampling strategy is as follows: k sub-models are sampled in each iteration, and the initial sub-model structure candidate set is Ω = {a s , a l}, where a s express The smallest submodel in a l express The largest submodel in , followed by random sampling The k-2 sub-models in the network are added to the sub-model structure candidate set Ω as the final sub-model candidate set for this iteration. The input feature of each iteration is x, which is input into the teacher network M teacher The soft label y = M is obtained teacher(x), and freeze its gradient y.detach(). Then traverse each sub-model a∈Ω in the sub-model structure candidate set Ω, and input the input feature x into the current sub-model to obtain the prediction vector Use this sub-model to predict the results Calculate the loss with the soft label y output by the teacher network KD represents the loss function, and performs gradient accumulation loss.backward() on the different sub-models sampled from Ω. After all sub-models in the candidate set Ω of the sub-model structure generated by each iteration are traversed, the model weights θ are updated uniformly. DST .

[0070] Furthermore, the model deployment and application described in step (9) are as follows:

[0071] If the current device computing resources are sufficient, the largest sub-model a is used. l Application, obtained through forward propagation at this time It has the best representation capability among the sub-models. When the computing resources of the device are not sufficient, the smallest sub-model a is used. s , obtained through forward propagation Because a s The amount of computation required is the smallest among all sub-models. At this time, the speed of forward propagation will be greatly improved, thereby improving the user experience. It also has good representation capabilities.

[0072] The bidirectional separable deep self-attention network can dynamically select sub-models of different sizes according to the computing resource status of the current device. Application achieves a dynamic balance between accuracy and latency, maintaining the accuracy of the sub-model while ensuring user experience.

[0073] The beneficial effects of the present invention are as follows:

[0074] The present invention proposes a bidirectional separable deep self-attention network, which is based on a bidirectional strategy that can be separable in both width and depth, and further selects reasonable sub-models using the "deep and narrow" filtering principle, and cooperates with the proposed self-distillation algorithm, so that each sub-model in the network has the application ability of visual question-answering tasks. This bidirectional separable deep self-attention model can dynamically select the appropriate sub-model to predict the answer based on the current computing resources, achieving a balance between accuracy and latency, and ensuring the accuracy of the predicted answer while giving users a good experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Schematic diagram of width and depth segmentation strategy according to an embodiment of the present invention;

[0076] Figure 2 Schematic diagram of sub-model filtering principle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The detailed parameters of the present invention are further described in detail below.

[0078] A lightweight visual question answering method based on a bidirectional separable deep self-attention network includes the following steps:

[0079] Step (1): Divide the data set;

[0080] The dataset described uses the VQA-v2 dataset, which is further divided into three subsets: training set, validation set, and test set. The training set is used to train the model, the validation set is used to locally verify the model convergence, and the test set is used for the final model performance evaluation.

[0081] Step (2): construct visual features of the image;

[0082] For a given image, the number m of candidate boxes in the image and their positions are detected using the existing trained object detection network; for each candidate box, the image corresponding to the candidate box area is input into the object detection network again, and the features before being input into the network classification layer are extracted as the features of the candidate box. The features extracted from each candidate box are then concatenated to form the visual features of the given image. In order to make the image feature dimension match the deep self-attention network, a learnable linear transformation is finally used to further process the image features and map them to the D-dimensional space; the specific method is as follows:

[0083] For a given image, the number m of candidate boxes and their positions in the image are inferred using the existing trained Faster R-CNN object detection network, and the image area corresponding to each candidate box is input into the Faster R-CNN object detection network to extract its visual features. For the i-th candidate box, its corresponding visual feature is The visual features corresponding to the entire image It is composed of the visual features corresponding to each candidate box. The specific formula is as follows:

[0084] X image =[x1, x2, ..., x i , ..., x m ] (Formula 1)

[0085] Then, a learnable linear transformation is used For image feature X image After further processing, it is mapped to D-dimensional space to obtain the final image visual features. The specific formula is as follows:

[0086] X input =Linear(X image ) (Formula 2)

[0087] Step (3): Construct the semantic features of the question;

[0088] For a given question, the trained word vector model is used to extract semantic features from each word in the question, and then the extracted word semantic features are concatenated to form the question semantic features. In order to make the question semantic feature dimension match the deep self-attention network, a learnable linear transformation is finally used to further process the question features and map them to a D-dimensional space; the specific method is as follows:

[0089] For a given question, which contains n words, each word is input into the pre-trained GloVe word vector model to extract its semantic features. For the jth word, its corresponding semantic feature is The semantic features corresponding to the entire question It is composed of the semantic features corresponding to each word. The specific formula is as follows:

[0090] Y question =[y1,y2,…,y j ,…,y n ] (Formula 3)

[0091] Then, a learnable linear transformation is used The question semantic feature Y question After further processing, it is mapped to D-dimensional space to obtain the final semantic features of the question. The specific formula is as follows:

[0092] Y input =Linear(Y question ) (Formula 4)

[0093] Step (4): Build a deep self-attention network;

[0094] The deep self-attention network is composed of multiple stacked self-attention layers, each of which is divided into two parts: a multi-head attention module and a feedforward layer. The deep self-attention network is used to construct a teacher network to guide the training and the final bidirectional separable deep self-attention network. The teacher network and the bidirectional separable deep self-attention network both use the same structure of the deep self-attention network. In order to make the input features match the dimensions of each sub-model in the bidirectional separable deep self-attention network, the deep self-attention network accepts image visual features of dimension D. and question semantic features As input, and through a linear projection transformation, the input features are mapped to d dimensions. The deep self-attention network can fully learn the interactive information between the two modalities and ultimately produce meaningful visual-semantic fusion features.

[0095] 4-1. Multi-head attention module;

[0096] For a given query feature Key Features And the value feature The multi-head attention module uses H parallel attention functions to calculate the features The specific formula is as follows:

[0097] F mha =MHA(Q,K,V)=[head1,head2,…,head H ]W 0 (Formula 5)

[0098]

[0099] in represents the mapping matrix of the h-th attention head, D H For the dimension of each attention head, we can use D H =D / H is calculated. In addition, It is used to further process the output features of the multi-head attention function. For the attention calculation method ATT, the specific formula is as follows:

[0100]

[0101] 4-2. Feedforward layer;

[0102] The feedforward layer consists of two layers of perceptrons, which perform nonlinear transformation on the output features of the multi-head attention module. Output Features The specific formula is as follows:

[0103]

[0104] in is the linear transformation projection matrix.

[0105] 4-3. Self-attention layer;

[0106] Each self-attention layer consists of the above-mentioned multi-head attention module and feed-forward layer. For a given input F input , output feature F output , the specific formula is as follows:

[0107]

[0108]

[0109] Where LN represents layer normalization.

[0110] 4-4. Stacking self-attention layers;

[0111] A deep self-attention network is composed of multiple stacked self-attention layers. The feature dimension does not change after the self-attention layer. Therefore, multiple self-attention layers can be connected in series to construct a deep self-attention network model. The specific formula is as follows:

[0112] Model = [Layer (1) , Layer (2) ,…,Layer (L) ] (Formula 11)

[0113] Where L is the number of self-attention layers.

[0114] Step (5): Design a width segmentation strategy;

[0115] Each self-attention layer in the deep self-attention network is composed of multiple parameter matrices. In order to adapt to input features of different dimensions, each parameter matrix needs to be split to match inputs of different dimensions and output output features of appropriate dimensions. For input features of dimension d, in order to maintain the original structural ratio of the self-attention layer, the output feature dimension is still d through the width splitting strategy. It is worth noting that for different sub-models with input features of different dimensions, the parameter matrix in the self-attention layer is shared. The smaller d is, the smaller the amount of shared parameters is; when d is equal to the original input dimension D, the parameter matrix is ​​not split;

[0116] For the parameter matrix W in multi-head attention Q , W K , And the input features of dimension d, keep the size of each attention head D H The input matching dimension D and the number of attention heads H of the corresponding parameter matrix are changed. The final segmented parameter matrix W Q , W K , in Indicates the number of attention heads after segmentation. The parameter matrix W in other self-attention layers 0 , W1, W2 adopt the same strategy, so that the final segmented parameter matrix

[0117] Step (6): Design a deep segmentation strategy;

[0118] The deep self-attention network is composed of multiple stacked self-attention layers, with the number of layers being L. When the number of layers of a sub-model is l<L, it is necessary to select the lth layer in the deep self-attention network according to the deep segmentation strategy and attribute it to the sub-model itself. The present invention proposes a simple and effective deep segmentation strategy. Under different layer settings, the sub-model can select the more important self-attention layer as much as possible, so that the final accuracy of different sub-models is improved;

[0119] For a deep self-attention network with L layers, the index of each layer is [1, 2, ..., L]. The present invention believes that the self-attention layers closer to the input and output are more important. This means that the middle layers are relatively less important. When the number of layers of the sub-model is l<L, it will start to discard the middle layers first. The specific operation is to sort each layer from the most important to the least important, and get the layer index as For a sub-model with l layers, take the first l items of the sorted layer index After reordering, restore to the original layer index order It is the depth segmentation strategy of the final l-layer sub-model.

[0120] Step (7): Combine the two segmentation strategies and design filtering principles;

[0121] Through the design of steps (5) and (6), each sub-model has a width d and a depth l. Under the same number of parameters and calculations, a deep and narrow sub-model will be more efficient and have a more reasonable structure than a shallow and wide sub-model. The present invention proposes a "deep and narrow" filtering principle, which selects some sub-models with many layers and low width before model training, and directly discards those sub-models with few layers and high width. Through this filtering principle, the candidate set of sub-model structures after screening is obtained. This filtering principle not only reduces the cost of model training, but also improves the accuracy of the trained sub-model;

[0122] For a given width ratio candidate set and the depth ratio candidate set By combining the candidate sets of these two dimensions, we get the preliminary sub-model structure candidate set Each sub-model structure in In order to easily express the "deep and narrow" filtering principle, a two-dimensional index matrix is ​​defined To further process the preliminary sub-model candidate set I(d, l) = 1 means that the sub-model a(d, l) is selected, and I(d, l) = 0 means that the sub-model a(d, l) is discarded. The index matrix I is first initialized to all 1 values, and then the lower triangular part is converted to 0 values. Finally, the selected sub-model The specific definitions are as follows:

[0123]

[0124] Step (8): Design a self-distillation training algorithm and train the model;

[0125] For the candidate set of sub-model structures obtained in step (7), A self-distillation training strategy is proposed to enable each sub-model to be fully trained. First, a teacher network is trained using the deep self-attention network in step (4), and a bidirectional separable deep self-attention network is constructed. When training the sub-models in the bidirectional separable deep self-attention network, the image and question are first input into the teacher network to obtain their prediction vectors, called soft labels, and the candidate set is sampled during training through the sub-model sampling strategy. The sub-model in the training set uses this soft label as the supervision label of the sampled sub-model for training;

[0126] Define the teacher network constructed by the deep self-attention network as M teacher , the bidirectional separable deep self-attention network is M DST , by training the teacher network M teacher Get its parameter weight θ and use this weight to initialize the bidirectional separable deep self-attention network M DST The weight θ DST . Through the sub-model sampling strategy, the candidate set is sampled during training The sub-model sampling strategy is as follows: k sub-models are sampled in each iteration, and the initial sub-model structure candidate set is Ω = {a s ,al}, where a s express The smallest submodel in a l express The largest submodel in , followed by random sampling The k-2 sub-models in the network are added to the sub-model structure candidate set Ω as the final sub-model candidate set for this iteration. The input feature of each iteration is x, which is input into the teacher network M teacher The soft label y = M is obtained teacher (x), and freeze its gradient y.detach(). Then traverse each sub-model a∈Ω in the sub-model structure candidate set Ω, and input the input feature x into the current sub-model to obtain the prediction vector Use this sub-model to predict the results Calculate the loss with the soft label y output by the teacher network KD represents the loss function, and performs gradient accumulation loss.backward() on the different sub-models sampled from Ω. After all sub-models in the candidate set Ω of the sub-model structure generated by each iteration are traversed, the model weights θ are updated uniformly. DST .

[0127] The model deployment and application described in step (9) are as follows:

[0128] If the current device computing resources are sufficient, the largest sub-model a is used. l Application, obtained through forward propagation Compared to the time It has the best representation capability among the sub-models. When the computing resources of the device are not sufficient, the smallest sub-model a is used. s , obtained through forward propagation Because a s The amount of computation required is the smallest among all sub-models. At this time, the speed of forward propagation will be greatly improved, thereby improving the user experience. It also has good representation capabilities.

[0129] In summary, the bidirectional separable deep self-attention network proposed in this invention can dynamically select sub-models of different sizes according to the computing resource status of the current device. Application achieves a dynamic balance between accuracy and latency, maintaining the accuracy of the sub-model while ensuring user experience.

[0130] like Figure 1 As shown in 2, the present invention proposes a lightweight visual question answering method based on a bidirectionally separable deep self-attention network.

[0131] The data set described in step (1) is divided as follows:

[0132] The final training set contains 115K images and 1.1M sentences, the validation set contains 5K images and 26K sentences, and the test set contains 80K images and 448K sentences.

[0133] The comprehensive features of the object in the constructed image described in step (2) are as follows:

[0134] For an image, it usually contains 36 candidate boxes, and the visual feature dimension extracted from each candidate box is 2048. The final mapped spatial dimension D will be adjusted accordingly according to the deep self-attention network. Taking D=512 as an example, the comprehensive feature of the image object obtained in this step is

[0135] The semantic features of the constructed question described in step (3) are as follows:

[0136] For a question, a fixed word length of 14 is usually set. Each word uses a pre-trained word vector model to extract its semantic feature dimension of 300. The final mapped spatial dimension D will be adjusted accordingly according to the deep self-attention network. Taking D=512 as an example, the semantic feature of the question obtained in this step is

[0137] The details of step (4) are as follows:

[0138] By setting D = 512, H = 8, input feature The feature F input Input into the multi-head attention module MHA and get the output Then the features Input into the feed-forward layer FFN to get the final output

[0139] The width segmentation strategy described in step (5) is as follows:

[0140] The present invention defines a set of splittable width ratio candidates as: Sub-model input feature dimensions under different width split ratios When D=512, the candidate width dimension is This means that there are four choices for the width dimension of the sub-model: 128, 256, 384, and 512.

[0141] The depth segmentation strategy described in step (6) is as follows:

[0142] The present invention defines the divisible depth ratio candidate set as The number of sub-model layers at different depth split ratios When L = 12, This means that there are four options for the number of sub-model layers: 2, 4, 8, and 12 layers.

[0143] The combination of the two segmentation strategies and the design of the filtering principle described in step (7) are as follows:

[0144] According to the width ratio candidate set defined in steps (5) and (6), and the depth ratio candidate set Combine the candidate sets of these two dimensions to obtain the candidate set of sub-model structure The final sub-model structure candidate set is obtained through filtering principle in

[0145] The self-distillation training algorithm described in step (8) is as follows:

[0146] The present invention sets k=4, which means that each iteration samples 1 largest sub-model, 1 smallest sub-model, and 2 other randomly sampled sub-models. In each iteration, 4 sub-models will be sampled and gradient accumulation will be performed together.

Claims

1. A lightweight visual question answering method based on a bidirectionally separable deep self-attention network, characterized by: It includes the following steps: Step (1): Divide the dataset; Step (2): Construct the visual features of the image; For a given image, use the existing trained object detection network to detect the number m of candidate boxes in the image and their positions; For each candidate box, input the image corresponding to the candidate box area into the object detection network again, and extract the features before the classification layer of the network as the features of the candidate box; Subsequently, splice the features extracted from each candidate box to form the visual features of the given image; in order to make the image feature dimension match the depth self-attention network, finally, use a learnable linear transformation to further process the image features and map them to the D-dimensional space; Step (3): Construct the semantic features of the question; For a given question, use the trained word vector model to extract the semantic features of each word in the question, and then splice the extracted single-word semantic features to form the question semantic features; in order to make the question semantic feature dimension match the depth self-attention network, finally, use a learnable linear transformation to further process the question features and map them to the D-dimensional space; Step (4): Construct the depth self-attention network; The depth self-attention network is stacked by multiple self-attention layers, and each self-attention layer is divided into two parts: the multi-head attention module and the feed-forward layer; use this depth self-attention network to construct the teacher network for guiding training and the final bidirectional splittable depth self-attention network; in order to make the input features match the dimensions of each sub-model in the bidirectional splittable depth self-attention network, this network accepts features of dimension D as input and maps the input features to the d-dimensional space through a linear projection transformation; Step (5): Design the width splitting strategy; Each self-attention layer in the depth self-attention network is composed of multiple parameter matrices. In order to adapt to input features of different dimensions, it is necessary to split each parameter matrix to match the input of different dimensions and output output features of appropriate dimensions; For the input features of dimension d, in order to maintain the original structure ratio of the self-attention layer, the width splitting strategy is used to make the output feature dimension still d; it should be noted that for different sub-models with input features of different dimensions, the parameter matrices in the self-attention layer are shared. When d is smaller, the shared number of parameters is smaller; when d is equal to the original input dimension D, the parameter matrix is not split; Step (13): Design the depth splitting strategy; The depth self-attention network is stacked by multiple self-attention layers. Denote the number of layers as L. When the number of layers l of the sub-model < L, select l layers in this depth self-attention network according to the depth splitting strategy and attribute them to the sub-model itself; Step (15): Combine the two splitting strategies and design the filtering principle; Through the design of steps (5) and (6), each sub-model has a width d and a depth l. Under the same number of parameters and calculations, a deep and narrow sub-model will be more efficient and have a more reasonable structure than a shallow and wide sub-model. A "deep and narrow" filtering principle is proposed. Before model training, some sub-models with many layers and low width are selected, and those with few layers and high width are directly discarded. Through this filtering principle, the candidate set of sub-model structures is obtained. Step (16): Design the self-distillation training algorithm and train the model; For the candidate set of sub-model structures obtained in step (7), A self-distillation training strategy is proposed to enable each sub-model to be fully trained. First, a teacher network is trained using the deep self-attention network in step (4), and a bidirectional separable deep self-attention network is constructed. When training the sub-models in the bidirectional separable deep self-attention network, the image and question are first input into the teacher network to obtain their prediction vectors, called soft labels, and the candidate set is sampled during training through the sub-model sampling strategy. The sub-model in the training set uses this soft label as the supervision label of the sampled sub-model for training; Step (17): Model deployment and application.

2. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 1, characterized in that: The division of the dataset described in Step (1) is specifically as follows: The dataset uses the VQA-v2 dataset, which is further divided into three subsets: a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to locally verify the convergence of the model, and the test set is used for the final model performance evaluation.

3. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 2, characterized in that: The visual features of the constructed image in step (2) are as follows: For a given image, use the existing trained Faster R-CNN object detection network to infer the number m of candidate boxes in the image and their positions, and input the image area corresponding to each candidate box into the Faster R-CNN object detection network to extract its visual features; For the i-th candidate box, its corresponding visual feature is The visual features corresponding to the entire image It is composed of the visual features corresponding to each candidate box. The specific formula is as follows: X image =[x1,x2,…,x i ,…,x m ] (Formula 1) Then, a learnable linear transformation is used For image feature X image After further processing, it is mapped to D-dimensional space to obtain the final image visual features. The specific formula is as follows: X input =Linear(X image ) (Formula 2).

4. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 3, characterized in that: The semantic features of the constructed question described in step (3) are as follows: For a given question, which contains n words, each word is input into the pre-trained GloVe word vector model to extract its semantic features; For the jth word, its corresponding semantic feature is The semantic features corresponding to the entire question It is composed of the semantic features corresponding to each word. The specific formula is as follows: Y question = [y1,y2,…,y i ,…,y n ] (Formula 3) Then, a learnable linear transformation is used The question semantic feature Y question After further processing, it is mapped to D-dimensional space to obtain the final semantic features of the question. The specific formula is as follows: Y input = Linear(Y question ) (Formula 4).

5. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 4, characterized in that: The construction of the deep self-attention network described in step (4) is as follows: The deep self-attention network is composed of multiple stacked self-attention layers, each of which is divided into two parts: a multi-head attention module and a feedforward layer; the deep self-attention network is used to construct a teacher network for guiding training and a final bidirectionally separable deep self-attention network, and the teacher network and the bidirectionally separable deep self-attention network both use a deep self-attention network with the same structure; in order to make the input features match the dimensions of each sub-model in the bidirectionally separable deep self-attention network, the deep self-attention network accepts image visual features of dimension D and question semantic features As input, the input features are mapped to d dimensions through a linear projection transformation; the deep self-attention network can fully learn the interactive information between the two modalities and ultimately generate meaningful visual-semantic fusion features; 4-1. Multi-head attention module; For a given query feature Key Features And the value feature The multi-head attention module uses H parallel attention functions to calculate the features The specific formula is as follows: F mha =MHA(Q,K,V)=[head1,head2,…,head H ]W 0 (Formula 5) in represents the mapping matrix of the h-th attention head, D H For the dimension of each attention head, D H =D / H is calculated; In addition, Used to further process the output features of the multi-head attention function; for the attention calculation method ATT, the specific formula is as follows: 4-2. Feedforward layer; The feedforward layer consists of two layers of perceptrons, which perform nonlinear transformation on the output features of the multi-head attention module. Output characteristics The specific formula is as follows: in is the linear transformation projection matrix; 4-3. Self-attention layer; Each self-attention layer consists of the above-mentioned multi-head attention module and feed-forward layer. For a given input F input , output feature F output , the specific formula is as follows: Where LN represents layer normalization; 4-4. Stacking self-attention layers; A deep self-attention network is composed of multiple stacked self-attention layers. The feature dimension does not change after the self-attention layer. Therefore, multiple self-attention layers are connected in series to construct a deep self-attention network model. The specific formula is as follows: Model=[layer (1) ,Layer (2) ,…,Layer (L) ] (Formula 11) Where L is the number of self-attention layers.

6. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 5, characterized in that: The width segmentation strategy described in step (5) is as follows: For the parameter matrix in multi-head attention And the input features of dimension d, keep the size of each attention head D H The input matching dimension D and the number of attention heads H of the corresponding parameter matrix remain unchanged, so that the parameter matrix finally divided in Represents the number of attention heads after segmentation; the parameter matrix W in other self-attention layers 0 , W1, W2 adopt the same strategy, so that the final segmented parameter matrix 7. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 6, characterized in that: The depth segmentation strategy described in step (6) is as follows: For a deep self-attention network with L layers, denote the index of each layer as [1, 2, …, L]. When the number of layers l of the sub-model is less than L, it will start discarding from the middle layer; the specific operation is to first sort the layers according to their importance from large to small, and obtain the layer indices as For the sub-model with l layers, take the first l items of the sorted layer indices Then, after re-sorting, restore to the original layer index order This is the depth slicing strategy for the final l-layer sub-model.

8. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 7, characterized in that: The two segmentation strategies described in step (7) are combined to design filtering principles, which are as follows: For a given width ratio candidate set and the depth ratio candidate set By combining the candidate sets of these two dimensions, we get the preliminary sub-model structure candidate set Each sub-model structure in In order to easily express the "deep and narrow" filtering principle, a two-dimensional index matrix is ​​defined To further process the preliminary sub-model candidate set I(d,l)=1 means that the sub-model a(d,l) is selected, and I(d,l)=0 means that the sub-model a(d,l) is discarded; the index matrix I is first initialized to all 1 values, and then the lower triangular part is converted to 0 values; finally, the selected sub-model The specific definitions are as follows:

9. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 8, characterized in that: The self-distillation training algorithm described in step (8) is as follows: Define the teacher network constructed by the deep self-attention network as M teacher , the bidirectional separable deep self-attention network is M DST , by training the teacher network M teacher Get its parameter weight θ and use this weight to initialize the bidirectional separable deep self-attention network M DST The weight θ DST ; Through the sub-model sampling strategy, sample the candidate set during training The sub-model sampling strategy is as follows: k sub-models are sampled in each iteration, and the initial sub-model structure candidate set is Ω = {a s ,a l }, where a s express The smallest submodel in a l express The largest submodel in , followed by random sampling The k-2 sub-models in the network are added to the sub-model structure candidate set Ω as the final sub-model candidate set for this iteration; the input feature of each iteration is x, which is input into the teacher network M teacher The soft label y = M is obtained teacher (x), and freeze its gradient y.detach(); then traverse each sub-model a∈Ω in the sub-model structure candidate set Ω, and input the input feature x into the current sub-model to obtain the prediction vector Use this sub-model to predict the results Calculate the loss with the soft label y output by the teacher network KD represents the loss function, and performs gradient accumulation loss.backward() on the different sub-models sampled from Ω. After all sub-models in the candidate set Ω of the sub-model structure generated by each iteration are traversed, the model weights θ are updated uniformly. DST .

10. The lightweight visual question answering method based on a bidirectionally separable deep self-attention network according to claim 9, characterized in that: The model deployment and application described in step (9) are as follows: If the current device computing resources are sufficient, the largest sub-model a is used. l Application, obtained through forward propagation at this time It has the best representation capability among the sub-models; when the computing resources of the device are not sufficient, the smallest sub-model a is used s , obtained through forward propagation Because a s The amount of computation required is the smallest among all sub-models. At this time, the speed of forward propagation will be greatly improved, thereby improving the user experience. Also has good representation ability; Bidirectionally separable deep self-attention network dynamically selects sub-models of different sizes according to the computing resource status of the current device Application achieves a dynamic balance between accuracy and latency, maintaining the accuracy of the sub-model while ensuring user experience.

Images