Visual question answering method and system based on attention mechanism combined with dense structure and transformer

Abstract

The present application relates to the technical field of computer vision, and especially relates to a visual question answering method and system based on attention mechanism combined with dense structure and Transformer, which comprises: analyzing the essential characteristics of visual question answering problems, and based on statistical analysis theory, performing mathematical modeling on the visual question answering problems; in view of the deficiency that the existing visual question answering methods do not consider the correlation information between images, a training data set initialization method based on YOLOv7 target detection model is designed; based on dense structure, attention mechanism and Transformer, a visual question answering model DensAttens-Trans is constructed; for the designed DensAttens-Trans model, a loss function based on cross-entropy loss is constructed, and the training data set after initialization is used to complete the training work of the DensAttens-Trans model, so that the computer system can understand and answer natural language questions about images. The present application gives a more robust, high fitting, real-time and human thinking text answer.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual question answering method and system based on attention mechanism combined with dense structure and Transformer. Background Technology

[0002] Visual Question Answering (VQA) is a classic cross-modal problem in computer vision. Its goal is to establish a mapping between images and text, enabling the visual system to intelligently answer natural language questions posed by users based on its understanding of a given image. With the widespread adoption of artificial intelligence across industries, VQA has found extensive applications in areas such as product recognition, intelligent education, and industrial inspection. Furthermore, VQA problems are frequently published in top journals and conferences in computer vision and artificial intelligence, such as CVPR, AAAI, and IJCV, demonstrating its significant research value in both practical applications and theoretical studies.

[0003] Existing visual question answering methods can be divided into two categories based on their processing principles: visual question answering methods based on generative models and methods based on visual language pre-trained models. Huang Q and Wei J proposed a dual-channel graph convolutional neural network (DC-GCN) in their paper "Aligned dualchannel graph convolutional network for visual question answering[C] / / Proceedings of the Annual Meeting of the Association for Computational Linguistics,2020:7166-7176". This graph convolutional neural network can capture the optical properties and imaging characteristics of visual targets in images, and can also obtain the dependencies between words and grammar in the question. Furthermore, it applies the principles of image alignment to natural language processing to achieve the goal of visual question answering. Lao M and Guo Y et al. proposed a multi-stage hybrid embedding fusion network (MHEF) in their paper “Multi-stage hybrid embedding fusion network for visual question answering[J].Neurocomputing,2021,423:541-550”. The proposed network uses a dual embedding fusion method to map one modality to another and uses a multi-stage fusion structure to fuse cross-modal features to achieve the goal of visual question answering. Cui Y and Yu Z et al. recognized that cross-modal associations between images and text affect the effectiveness of visual question answering. In their paper "ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross-andIntra-modal Knowledge Integration[C] / / Proceedings of the 29th ACM International Conference on Multimedia,2021:797-806", they proposed a method based on a pre-trained model of visual language processing, which they named ROSITA. The proposed method introduces a knowledge masking strategy and uses the scene graph structure as prior knowledge to eliminate interference information within and between modalities, thereby achieving the final visual question answering function.In their paper "Improving visual question answering for bridge inspection by pre-training with external data of image-text pairs[J]. Computer-Aided Civil and Infrastructure Engineering, 2024, 39(3): 345-361", Kunlamai T and Yamane T et al. mainly enhanced the image-text training dataset, obtaining a larger and higher-quality dataset, and combined multimodal and deep learning to construct models, thereby improving the accuracy of existing visual question answering methods. In their paper "Robust Visual Question Answering: Datasets, Methods, and Future Challenges[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024: 1-20", Ma J and Wang P et al. mainly explored the future development direction of visual question answering methods. First, they measured the correlation between different methods, then estimated the evaluation indicators with high usage of existing methods, and finally concluded that future visual question answering methods need to have stronger robustness.

[0004] However, existing visual question answering methods all use pre-trained models and synthetic datasets for training, which leads to problems such as underfitting, model incompatibility, and answers that do not conform to human thinking. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing visual question answering methods in terms of search space, universality, and robustness. It proposes a visual question answering method and system based on attention mechanism combined with dense structure and Transformer. Based on cross-modal information of image and text, it fully mines the features contained in the image and the semantic information in the text. While fully analyzing the image, it combines the semantic relationship of the user's question to provide a more robust, well-fitting, and human-like text answer.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a visual question answering method based on an attention mechanism combined with dense structures and Transformers, comprising:

[0008] This paper analyzes the essential characteristics of visual question answering problems and, based on statistical analysis theory, performs mathematical modeling of visual question answering problems.

[0009] The YOLOv7 object detection model is used to obtain the object detection results corresponding to each training image. The obtained object detection results are used as labels and added to each training image, question and answer to complete the initialization of the training dataset.

[0010] The optical properties and imaging characteristics of various images in the initialized training dataset are analyzed, and an image feature encoding module is constructed based on channel attention. Based on the semantic and structural information of natural language questions in the initialized training dataset, a question feature encoding module that integrates dense structure and Transformer encoding structure is constructed. Combining the image feature encoding results and the question feature encoding results, a feature decoding module is constructed. By integrating the image feature encoding module, the question feature encoding module, and the feature decoding module, a visual question answering model DensAttens-Trans is constructed based on a feature selection scheme.

[0011] By combining the initial training dataset, the DensAttens-Trans model is trained using a constructed loss function based on cross-entropy loss. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer from a given image and a natural language question.

[0012] According to the visual question answering method based on the attention mechanism combined with dense structure and Transformer of the present invention, the mathematical model of the visual question answering problem is further defined as follows:

[0013] answer=argmaxP(ans|(I,Q);θ)

[0014] In the formula, I is the given image to be asked, Q is the question posed to the image, θ is the parameters of the constructed model after training, ans is one or more possible answers obtained from the model, P(·) is the probability value corresponding to each answer, and answer is the answer with the highest probability value among the answers ans obtained by combining image I and question Q.

[0015] According to the visual question answering method based on the attention mechanism combined with dense structure and Transformer of the present invention, the initialization process of the training dataset is further as follows:

[0016] First, the object detection results in the images obtained using YOLOv7 are saved in text form and added as tags to the external storage module of the training dataset images. Then, a "*end / " marker is added to the end of the natural language strings of the questions and answers, and the saved YOLOv7 object detection result text is added after the "*end / " marker in the questions and answers.

[0017] According to the visual question answering method based on attention mechanism combined with dense structure and Transformer of the present invention, the image feature encoding module constructed based on channel attention further includes: first, constructing an image feature encoding operation unit based on 3×3 convolution, channel attention and 2×2 average pooling; second, superimposing 4 constructed image feature encoding operation units to form an image feature encoding module.

[0018] According to the visual question answering method based on attention mechanism and combined dense structure and Transformer of the present invention, the construction of question feature encoding module that integrates dense structure and Transformer encoding structure includes: first, constructing 5 question feature encoding units based on the encoding module in the Transformer model; second, densely connecting the 5 question feature encoding units based on the dense structure connection method in DenseNet to obtain the question encoded features.

[0019] According to the visual question answering method based on attention mechanism combined with dense structure and Transformer of the present invention, the feature decoding module is further constructed by: first, based on the question feature encoding unit, adding a fusion self-attention module and two modules, feature fusion and normalization, to the front end of the self-attention module of the question feature encoding unit to construct a decoding unit; then, superimposing five decoding units with the same structure to form a feature decoding module.

[0020] According to the visual question answering method based on attention mechanism combined with dense structure and Transformer of the present invention, the construction of the visual question answering model DensAttens-Trans further includes: First, based on the designed image feature encoding module and question feature encoding module, and based on a parallel architecture, the image and question feature encoding modules in the DensAttens-Trans model are constructed respectively; Second, based on the designed feature decoding module, and based on a multi-branch structure, the feature decoding module in the DensAttens-Trans model is constructed; Finally, after the feature decoding module, a feature selector and a Softmax classifier are designed to complete the construction of the DensAttens-Trans model; wherein, the question-encoded features are input one by one into each decoding unit, the first decoding unit processes the image-encoded features and the question-encoded features, and subsequent decoding units process the output of the previous decoding unit and the question-decoded features.

[0021] According to the visual question answering method based on the attention mechanism combined with dense structure and Transformer of the present invention, the expression of the loss function based on cross-entropy loss is further as follows:

[0022]

[0023] Where L is the loss function, M is the number of questions in the training data, N is the number of answers in the training data, i and j are the indices of questions and answers in the training dataset, respectively, and S... ij It is a binary real label. D is the probability value of the output answer based on the classification. t These are labels for object detection. T is the probability value of the detected object, T is the number of object types contained in YOLOv7, and t is the index of the object detection result using YOLOv7.

[0024] Furthermore, the present invention also provides a visual question answering system based on an attention mechanism combined with dense structures and Transformers, for implementing the visual question answering method based on an attention mechanism combined with dense structures and Transformers as described above. The system includes:

[0025] The Visual Question Answering Problem Modeling Unit is used to analyze the essential characteristics of visual question answering problems and to perform mathematical modeling of visual question answering problems based on statistical analysis theory.

[0026] The data preprocessing unit is used to obtain the target detection results corresponding to each training image using the YOLOv7 object detection model, and add the obtained target detection results as labels to each training image, question and answer to complete the initialization of the training dataset.

[0027] The visual question answering model building unit is used to analyze the optical properties and imaging characteristics of various images in the initialized training dataset, and construct an image feature encoding module based on channel attention; based on the semantic and structural information of natural language questions in the initialized training dataset, a question feature encoding module that integrates dense structure and Transformer encoding structure is constructed; combining the image feature encoding results and the question feature encoding results, a feature decoding module is constructed; and by integrating the image feature encoding module, the question feature encoding module, and the feature decoding module, a visual question answering model DensAttens-Trans is constructed based on a feature selection scheme.

[0028] The loss function calculation unit is used to train the DensAttens-Trans model using the constructed cross-entropy loss-based loss function in conjunction with the initialized training dataset. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer to a given image and a natural language question.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. Existing visual question answering datasets suffer from insufficiently strong non-linear relationships between images, questions, and answers, resulting in low efficiency for existing visual question answering methods. To address this issue, YOLOv7 is introduced into visual question answering, establishing non-linear relationships between images, questions, and answers in the dataset. This narrows the search space for the correspondence between the posed question and the given answer, thereby improving the real-time performance, robustness, and applicability of visual question answering methods.

[0031] 2. This invention integrates the dense structure of DenseNet with the encoding module of the Transformer model to construct a question feature encoding module, thereby more fully mining the semantic information in the text and more effectively encoding the question features. Channel attention and self-attention mechanisms are introduced into the visual question answering model to fully mine the features in the image, ensuring effective encoding of image features while simultaneously decoding both the encoded image features and the encoded question features.

[0032] In summary, the novel visual question answering model DensAttens-Trans constructed in this invention can effectively process visual question answering questions, providing more robust, well-fitting, and human-thinking text answers, and search for answers more efficiently and with better universality. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the visual question answering method based on attention mechanism combined with dense structure and Transformer according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the image feature encoding module structure according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the problem feature encoding module structure according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the feature decoding module structure according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the DensAttens-Trans model structure according to an embodiment of the present invention;

[0039] Figure 6 This is a structural block diagram of a visual question-answering system based on attention mechanism combined with dense structure and Transformer according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] To address the shortcomings of existing visual question answering methods in terms of search space, universality, and robustness, this invention provides a visual question answering method based on an attention mechanism combined with dense structures and Transformers, such as... Figure 1 As shown, it includes the following steps:

[0042] Step S101: Determine the academic concept of visual question answering: Analyze the essential characteristics of visual question answering and, based on statistical analysis theory, perform mathematical modeling of visual question answering.

[0043] Visual question answering (VQA) involves a visual system providing an answer to a given image and a question. The general approach to VQA is to search a specified answer space and determine the answer with the highest probability value based on the search results. Based on this analysis, the mathematical model for visual question answering is defined as follows:

[0044] answer=argmax P(ans|(I,Q);θ)

[0045] In the formula, I is the given image to be asked, Q is the question posed to the image, θ is the parameters of the constructed model after training, ans is one or more possible answers obtained from the model, P(·) is the probability value corresponding to each answer, and answer is the answer with the highest probability value among the answers ans obtained by combining image I and question Q. Analysis of the above formula shows that the key to solving the visual question answering problem is to determine the parameters θ after model training. The structure and training process of the visual question answering model are important factors affecting the value of parameter θ. Therefore, this invention will focus on describing the construction and training process of the visual question answering model.

[0046] Step S102: Initialize the training dataset: To address the shortcomings of existing visual question answering methods that do not consider the correlation information between images, the YOLOv7 object detection model is used to obtain the object detection results corresponding to each training image. The obtained object detection results are then added as labels to each training image, question, and answer, thereby completing the initialization of the training dataset.

[0047] Existing visual question answering datasets consist of three parts: an image database, a question space, and an answer space. However, these datasets suffer from insufficient non-linear relationships between images, questions, and answers, leading to low efficiency in answer search for existing visual question answering methods. To address this issue, the YOLOv7 object detection model is introduced into the visual question answering method to establish non-linear relationships between images, questions, and answers in the dataset, thereby improving the method's real-time performance, robustness, and applicability.

[0048] The initialization process is as follows: First, the object detection results in the images obtained using YOLOv7 are saved in text form and added to the external storage module of the training dataset as tags; then, a "*end / " marker is added to the end of the natural language strings of the question and answer, and the saved YOLOv7 object detection result text is added after the "*end / " marker in the question and answer. This greatly reduces the scope when searching for answers and improves search efficiency.

[0049] Step S103: Construct the visual question answering model DensAttens-Trans.

[0050] As can be seen from the mathematical model of visual question answering, the key to visual question answering lies in constructing and training the visual question answering model. Inspired by the success of large models such as Transformer in the field of natural language processing, the proposed method improves the Transformer model and constructs the visual question answering model DensAttens-Trans. The constructed DensAttens-Trans consists of four parts: an image feature encoding module, a question feature encoding module, a feature decoding module, and a classifier. Among them, the classifier structure is relatively simple, so the structure and working principle of the image feature encoding module, question feature encoding module, and feature decoding module will be introduced in detail.

[0051] ① Image feature coding module

[0052] Since the images processed in visual question answering problems are generally ideal visible light images, a superimposed feature encoding structure is used to extract image features to improve the real-time performance of the method. The specific structure of the image feature encoding module is as follows: First, 3×3 convolution is used to calculate and reuse the channels of the input image, increasing the number of image channels while reshaping the image features. Channel attention is used to select key channels in the image features, reducing the dimensionality of the extracted features. 2×2 average pooling is used to reduce the scale of the image features, further reducing the feature size. Second, image feature encoding operation units are constructed using 3×3 convolution, channel attention, and 2×2 average pooling. Four constructed image feature encoding operation units are superimposed to form the image feature encoding module. The structure of the constructed image feature encoding module is as follows: Figure 2 As shown.

[0053] The image feature encoding module encodes the image as follows: First, the image to be queried is input to the first image feature encoding operation unit; then, the output features of the previous image feature encoding operation unit are input to the next image feature encoding operation unit; finally, the features output by the fourth image feature encoding operation unit are the encoded features of the image; the encoding process of the image feature encoding module can be represented as:

[0054] Ifeature1=avg_p(CA(Conv(I)))

[0055] Ifeature i =avg_p(CA(Conv(Ifeature) i-1 )))sti∈N + ,i∈[2,4]

[0056] Ifeature encoder =Ifeature4

[0057] Where I is the input image from which features are to be extracted, Conv(·) is a 3×3 convolution, CA(·) is channel attention, avg_p(·) is 2×2 average pooling, and Ifeature i It is the feature output by the i-th image feature encoding operation unit, Ifeature encoder It is the feature output by the 4th image feature coding operation unit, that is, the feature output by the image feature coding module.

[0058] ② Problem Feature Encoding Module

[0059] In visual question answering, the analysis of the natural language question posed by the image is crucial for obtaining an accurate answer. Natural language questions possess strong semantic and contextual characteristics, making question feature extraction paramount. Given the good performance of Transformer in natural language processing and DenseNet's superior performance in extracting structured semantic information, a question feature encoding module integrating dense structure and Transformer encoding structure was designed. The specific structure is as follows: First, based on the encoding module in the Transformer model, five question feature encoding units are constructed; second, based on the dense connection method in DenseNet, the five question feature encoding units are densely connected to obtain the encoded question features. The structure of the constructed question feature encoding module is as follows: Figure 3 As shown.

[0060] The problem feature encoding module encodes the problem as follows: First, the proposed problem is input into the first problem feature encoding unit; then, following the principle of dense connections in DenseNet, the i-th problem feature encoding unit needs to encode the outputs of the previous i-1 problem feature encoding units simultaneously. That is, the third problem feature encoding unit needs to process the outputs of the first and second units, and the fourth problem feature encoding unit needs to process the outputs of the first, second, and third units; finally, the feature output by the fifth problem feature encoding unit is the encoded feature of the problem. The encoding process of the problem feature encoding module can be represented as:

[0061]

[0062]

[0063] Qfeature encoder =Qfeature5

[0064] Where Q represents the feature to be extracted, conv1(·) is a 1×1 convolution, and Qfeature i It is the feature output by the i-th problem feature encoding unit, Qfeature encoder These are the features that are finally extracted. It is a feature fusion and normalization operation, and η(·) is the feedforward processing. It is a matrix multiplication operation. It is a feature dimension fusion operation.

[0065] ③ Feature decoding module

[0066] After encoding the image and problem features, the next task, according to the Transformer's operating mechanism, is to decode these two types of encoded features. This method constructs a feature decoding module based on the Transformer feature decoding structure, with the following specific structure: First, based on the problem feature encoding unit, a fusion self-attention module and two modules (feature fusion and normalization) are added to the front of the self-attention module of the problem feature encoding unit to construct a decoding unit; then, five decoding units with the same structure are superimposed to form the feature decoding module. The structure of the constructed feature decoding module is as follows: Figure 4 As shown.

[0067] The operation process of the feature decoding module is as follows: First, the image is encoded into features. encoder Q-feature after problem encoding encoder The features are input into the self-attention fusion module, and the output features are fused and normalized. Next, the fused and normalized features are processed according to the self-attention module's operational rules. Finally, the above steps are repeated four times to output the decoded features. The operation process of the feature decoding module can be represented as follows:

[0068]

[0069] Fdecoder i =ψ(Fdecoder i-1 )sti∈N + ,i∈[2,5]

[0070] Fdecoder = Fdecoder5

[0071] In the two formulas above, Fdecoder i ψ(·) is the feature map output by the i-th decoding unit, φ(·) is the fusion self-attention function, Fdecoder is the feature output by the feature decoding module, and ψ(·) is the decoding unit operation function.

[0072] ④ Construct the DensAttens-Trans model

[0073] After constructing the image feature encoding module, question feature encoding module, and feature decoding module, the next step is to integrate these modules and construct the visual question answering model DensAttens-Trans based on a feature selection scheme. The specific method is as follows: First, based on the designed image feature encoding module and question feature encoding module, and using a parallel architecture, construct the image and question feature encoding modules in the DensAttens-Trans model respectively. Second, based on the designed feature decoding module, and using a multi-branch structure, construct the feature decoding module in the DensAttens-Trans model. Finally, after the feature decoding module, design a feature selector and a softmax classifier to complete the construction of the DensAttens-Trans model. The structure of the constructed DensAttens-Trans model is as follows: Figure 5 As shown.

[0074] The DensAttens-Trans model works as follows: First, the image and question are input in parallel into the image feature encoding module and question feature encoding module of the DensAttens-Trans model to obtain the encoded features of the image and question, respectively. Second, the encoded features of the image are directly input into the first decoding unit of the feature decoding module, while the extracted encoded features of the question are sequentially input into each decoding unit. At this point, the first decoding unit processes the encoded features of the image and question, and subsequent decoding units process the output of the previous decoding unit and the encoded features of the question. Finally, the initial cross-modal decoded features output from the fifth decoding unit are selected to form the final cross-modal decoded features. These final cross-modal decoded features are then input into the Softmax classifier to obtain the final answer to the question about the image.

[0075] Step S104: Construct a loss function based on cross-entropy loss: Combine the initialized training dataset and use the constructed loss function based on cross-entropy loss to train the DensAttens-Trans model. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer from a given image and a natural language question.

[0076] After completing the DensAttens-Trans model, to obtain the mapping relationship between images, questions, and answers, the DensAttens-Trans model needs to be trained using the initialized training data. Since the structure of the original training data has been modified, object detection labels are added to the existing cross-entropy loss function. The constructed loss function can be expressed as:

[0077]

[0078] Where L is the loss function, M is the number of questions in the training data, N is the number of answers in the training data, i and j are the indices of questions and answers in the training dataset, respectively, and S... ij It is a binary real label. D is the probability value of the output answer based on the classification. t These are labels for object detection. T is the probability value of the detected object, T is the number of object types contained in YOLOv7, and t is the index of the object detection result using YOLOv7.

[0079] Furthermore, based on the above method, embodiments of the present invention also provide a visual question-answering system based on an attention mechanism combined with dense structures and Transformers, such as... Figure 6 As shown, it includes a visual question answering problem modeling unit, a data preprocessing unit, a visual question answering model building unit, and a loss function calculation unit, wherein,

[0080] The Visual Question Answering Problem Modeling Unit is used to analyze the essential characteristics of visual question answering problems and to perform mathematical modeling of visual question answering problems based on statistical analysis theory.

[0081] The data preprocessing unit is used to obtain the target detection results corresponding to each training image using the YOLOv7 target detection model, and add the obtained target detection results as labels to each training image, question and answer to complete the initialization of the training dataset.

[0082] The visual question answering model building unit is used to analyze the optical properties and imaging characteristics of various images in the initialized training dataset and construct an image feature encoding module based on channel attention. Based on the semantic and structural information of natural language questions in the initialized training dataset, a question feature encoding module that integrates dense structure and Transformer encoding structure is constructed. Combining the image feature encoding results and the question feature encoding results, a feature decoding module is constructed. By integrating the image feature encoding module, the question feature encoding module, and the feature decoding module, a visual question answering model DensAttens-Trans is constructed based on a feature selection scheme.

[0083] The loss function calculation unit is used to train the DensAttens-Trans model using the constructed cross-entropy loss-based loss function in conjunction with the initialized training dataset. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer to a given image and a natural language question.

[0084] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0086] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0087] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0088] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual question answering method based on attention mechanism combined with dense structure and Transformer, characterized in that, Include: S1: Analyze the essential characteristics of visual question answering problems and, based on statistical analysis theory, perform mathematical modeling of visual question answering problems; S2: Use the YOLOv7 object detection model to obtain the object detection results corresponding to each training image, and add the obtained object detection results as labels to each training image, question and answer to complete the initialization of the training dataset; S3: Analyze the optical properties and imaging characteristics of various images in the training dataset after initialization, and construct an image feature encoding module based on channel attention; Based on the semantic and structural information of natural language questions in the initial training dataset, a question feature encoding module that integrates dense structure and Transformer encoding structure is constructed; a feature decoding module is constructed by combining image feature encoding results and question feature encoding results; and a visual question answering model DensAttens-Trans is constructed by integrating the image feature encoding module, question feature encoding module and feature decoding module based on a feature selection scheme. The problem feature encoding module that integrates dense structure and Transformer encoding structure is constructed as follows: First, based on the encoding module in the Transformer model, five problem feature encoding units are constructed; second, based on the dense structure connection method in DenseNet, the five problem feature encoding units are densely connected to obtain the problem encoded features. The construction of the feature decoding module includes: First, based on the problem feature encoding unit, two modules, namely a fusion self-attention module and a feature fusion and normalization module, are added to the front end of the self-attention module of the problem feature encoding unit to construct the decoding unit; then, five decoding units with the same structure are superimposed to form the feature decoding module. The construction of the visual question answering model DensAttens-Trans includes: First, based on the designed image feature encoding module and question feature encoding module, image and question feature encoding modules in the DensAttens-Trans model are constructed using a parallel architecture. Second, based on the designed feature decoding module, feature decoding modules in the DensAttens-Trans model are constructed using a multi-branch structure. Finally, after the feature decoding module, a feature selector and a Softmax classifier are designed to complete the construction of the DensAttens-Trans model. The question-encoded features are input into each decoding unit sequentially. The first decoding unit processes the image-encoded features and the question-encoded features, while subsequent decoding units process the output of the previous decoding unit and the question-decoded features. S4: Combine the initialized training dataset and train the DensAttens-Trans model using the constructed loss function based on cross-entropy loss. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer from a given image and a natural language question.

2. The visual question answering method based on attention mechanism combined with dense structure and Transformer according to claim 1, characterized in that, The mathematical model for visual question answering is defined as follows: In the formula, The given image is the one to be asked. The question is directed at images. These are the parameters after the constructed model has been trained. It is based on one or more possible answers obtained from the model. These are the probability values ​​corresponding to each answer. It combines images With raising questions The answer obtained The answer with the highest probability value.

3. The visual question answering method based on attention mechanism combined with dense structure and Transformer according to claim 1, characterized in that, The initialization process for the training dataset is as follows: First, the object detection results in the images obtained using YOLOv7 are saved in text form and added as tags to the external storage module of the training dataset images. Then, "*end / " is added to the end of the natural language strings of the questions and answers, and the saved YOLOv7 object detection results text is added after the "*end / " tags in the questions and answers.

4. The visual question answering method based on attention mechanism combined with dense structure and Transformer according to claim 1, characterized in that, The image feature coding module based on channel attention is constructed as follows: First, an image feature coding operation unit based on 3×3 convolution, channel attention, and 2×2 average pooling is constructed; second, four constructed image feature coding operation units are superimposed to form an image feature coding module.

5. The visual question answering method based on attention mechanism combined with dense structure and Transformer according to claim 1, characterized in that, The constructed loss function based on cross-entropy loss is expressed as follows: ,in, It is a loss function. It is the number of questions in the training data. It is the number of answers in the training data. and These are the indices of the questions and answers in the training dataset. It is a binary real label. It is the probability value of the output answer based on the category. These are labels for object detection. It is the probability value of object detection. It utilizes the types of targets contained within YOLOv7, and t is the index of the target detection results using YOLOv7.

6. A visual question-answering system based on an attention mechanism combined with dense structure and Transformer, characterized in that, For implementing the visual question answering method based on attention mechanism combined with dense structure and Transformer as described in any one of claims 1-5, the system comprises: The Visual Question Answering Problem Modeling Unit is used to analyze the essential characteristics of visual question answering problems and to perform mathematical modeling of visual question answering problems based on statistical analysis theory. The data preprocessing unit is used to obtain the target detection results corresponding to each training image using the YOLOv7 object detection model, and add the obtained target detection results as labels to each training image, question and answer to complete the initialization of the training dataset. The visual question answering model building unit is used to analyze the optical properties and imaging characteristics of various images in the initialized training dataset and construct an image feature encoding module based on channel attention. Based on the semantic and structural information of natural language questions in the initial training dataset, a question feature encoding module that integrates dense structure and Transformer encoding structure is constructed; a feature decoding module is constructed by combining image feature encoding results and question feature encoding results; and a visual question answering model DensAttens-Trans is constructed by integrating the image feature encoding module, question feature encoding module and feature decoding module based on a feature selection scheme. The loss function calculation unit is used to train the DensAttens-Trans model using the constructed cross-entropy loss-based loss function in conjunction with the initialized training dataset. The trained DensAttens-Trans model can then be used to achieve visual question answering, which obtains an answer to a given image and a natural language question.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-5.

Images